Method of remotely controlling external services and selectively sharing control of the same

ABSTRACT

A method of controlling an external service among a plurality of external services, by receiving an input command at a mobile device, translating the input command into a common language sharable between the external services so that such input commands are understood between the plurality of devices and services and functions are performed at the devices and or services in response to the input command. This control of the services is also sharable as instructed by an input command.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. §120 from U.S. patentapplication Ser. No. 14/738,121, filed on Jun. 12, 2015, which claimspriority from U.S. patent application Ser. No. 14/608,786, filed on Jan.29, 2015, issued on Aug. 18, 2015 as U.S. Pat. No. 9,111,214, whichclaims priority to Provisional Application No. 61/933,405, filed on Jan.30, 2014, in the United States Patent and Trademark Office, thedisclosures of which are incorporated herein in their entirety byreference.

BACKGROUND OF THE INVENTIVE CONCEPT

1. Field of the Invention

The present general inventive concept relates to a virtual assistantsystem to wirelessly control a plurality of external devices andservices and to provide communication and sharing of information betweenthe plurality of external devices and services, and to selectively allowsharing of the control with other users.

2. Description of the Related Art

Advancements in wireless communications have allowed innovation in datasharing to grow at a rapid rate. A user may use a mobile device (such asa phone, tablet computer, wristwatch, etc.) to open up variousapplications/programs, operate various devices, and communicate withother mobile devices. A development called the Internet of Things, orIoT, reflects a growing trend of connecting to and controlling variousdevices and services wirelessly.

However, the mobile device is limited to requiring the user to inputcommands, instructions, and data in a manner unique to each specificapplication/program and external devices. More specifically, programssuch as Facebook™ and LinkedIn™ each communicate in their ownproprietary languages. If a user wishes to look up a person's profile inLinkedIn™ after having found that person in Facebook™, the user isrequired to type in proper search criteria in proper fields within theLinkedIn™ program itself. As such, although a user may separatelycommunicate with Facebook™ and LinkedIn™, these programs do not andcannot communicate with each other.

As another example, Phillips has developed a light emitting diode (LED)light bulb (i.e., the Phillips “Hue”) that allows a user to changecolors emitting therefrom via a mobile device that is running anapplication associated with the bulb. More specifically, the user mustdownload and install, onto the mobile device, the application associatedwith the bulb, which communicates with the bulb using the Phillips Hueproprietary language. After the application is installed and running onthe mobile device, the user may then control the tone, contrast, andcolor of the bulb using the mobile device. However, the user cannotgrant access to the light bulb to another user operating another mobiledevice. Instead, if the other user wishes to control the light bulb withthe other mobile device, the other user must also download and installthe application associated with the bulb into the other mobile device.

Moreover, the user cannot use information within the applicationassociated with the light bulb to interact with another application. Forexample, the user may not use information within the light bulbapplication to purchase more light bulbs in another application such asAmazon™. As such, the application associated with the bulb is limitedmerely to allowing a user to control the particular light bulbassociated with the application stored within the user's mobile device.

Further, Apple™ and Google™ each include a voice assistant (Siri forApple and Google Now for Google) on its respective mobile device thattranslates a voice received into a search program. However, the Siri™and Google Now™ assistants are limited to providing a search and somedevice actions, and have a limited integration with other applicationssuch as OpenTable™, etc.

Accordingly, there is a need for a system that allows all devices,programs, services and applications connected with the system tocommunicate therebetween.

Further, there is a need to be able to control any external devices orservices desired via the web by inputting commands to perform suchcontrols from a hand-held device.

SUMMARY OF THE INVENTIVE CONCEPT

The present general inventive concept provides a method of controlling aplurality of external services and selectively sharing control of thesame.

Additional features and utilities of the present general inventiveconcept will be set forth in part in the description which follows and,in part, will be obvious from the description, or may be learned bypractice of the general inventive concept.

The foregoing and/or other features and utilities of the present generalinventive concept may be achieved by providing a method of controllingan external service among a plurality of external services, the methodincluding receiving an input command at a mobile device, the inputcommand corresponding to a function to be performed at the externalservice, translating the input command into a semantic informationinterchange format language at the mobile device, receiving thetranslated input command at a server device wirelessly connected to theplurality of external services, obtaining a proprietary language of theexternal service to be controlled at the server device, the obtainedproprietary language corresponding to the translated input command, anddetermining whether to perform the function at the external serviceusing the obtained proprietary language at one of the server device orthe mobile device based on predetermined conditions.

In an exemplary embodiment, the method may further include performingthe function at the external device by the mobile device when it isdetermined that a local network connection is established between themobile device and the external service, otherwise performing thefunction at the external device by the server device through thewireless connection.

In an exemplary embodiment, if it is determined that a local networkconnection is established between the mobile device and the externalservice, the method further includes translating the obtainedproprietary language of the input command into the semantic informationinterchange format language at the server device, transmitting thetranslated proprietary language to the mobile device, and converting thetransmitted semantic information interchange format language inputcommand back to the proprietary language at the mobile device.

In an exemplary embodiment, the method may further include analyzingcontext information of the input command at the server device to obtainhistory of use data of the input command.

In an exemplary embodiment, the context information may include at leastone of a location of the mobile device when the input command isreceived, a time of day when the input command is received, co-presenceof specific individuals, involuntary actions by a user, and actionsequences taken by a user of the mobile device.

In an exemplary embodiment, the history of use data may identify atleast one specific user, and the received input command may be convertedaccording to the at least one specific identified user.

In an exemplary embodiment, the method may further include authorizingat least one other user to control the external service through inputcommands according to the history of use data. The history of use datamay include data identifying a history of providing authorization toother users to control the external service through input commandsaccording to stored credentials.

In an exemplary embodiment, if it is determined that there are aplurality of possible intended functions to be performed by the inputcommand at the server device, the method may further include displayingthe possible intended functions to be performed at the mobile device tobe selected as a confirmation input command, and updating a storedlibrary of recognizable input commands according to the selectedconfirmation input command received.

In an exemplary embodiment, the method may further include recognizing arepeated cluster of input commands at the server device, and creating anew consolidated input command within a stored library of semanticinformation interchange format commands to represent performing thecluster of input commands simultaneously.

In an exemplary embodiment, the cluster of commands may include a firstinput command that is invalidated and a second input commandsubsequently input, and updating the library may include associating thesecond input command with the function previously associated with thefirst input command.

In an exemplary embodiment, method may further include, if the inputcommand is not recognized, requesting an alternate input command at themobile device, and updating a stored library of semantic informationinterchange format commands with the received alternate input command.

In an exemplary embodiment, the method may further include storingcredentials at the server device, the credentials corresponding to auser and including data to authorize the user to control the externalservice through the input commands, and updating the stored credentialsat the server device according to a received input command to authorizeat least one other user to control the external service through inputcommands.

In an exemplary embodiment, the method may further include revoking theauthorization of the at least one other user to control the externalservice at the server device according to another input command.

In an exemplary embodiment, the receiving the other input command torevoke the authorization may include displaying a list of authorizedusers at the mobile device, and selecting the at least one other user tohave access revoked.

In an exemplary embodiment, the method may further include connectingthe server device to the plurality of external services by presentingthe stored credentials to the respective external services anddetermining whether further authentication is necessary to inputcommands to the services, displaying a request for input of additionalcredentials if it is determined that further authentication isnecessary, and updating the stored credentials with the additionalcredentials received.

In an exemplary embodiment, a non-transitory computer-readable mediummay contain computer-readable codes to perform the method.

The foregoing and/or other features and utilities of the present generalinventive concept may also be achieved by providing a method ofcontrolling an external service among a plurality of external services,the method including storing credentials of a user at a server device,receiving at a mobile device an input command to perform a function atthe external service to be controlled, translating the received inputcommand to a semantic information interchange format language at themobile device, determining at the server device whether the storedcredentials authorize the user to input commands to control the externalservice, converting, at the server device, the translated input commandto a corresponding input command understood by the external service whenthe credentials authorize the user to input commands, and controllingthe external service with the input command understood by the externalservice.

In an exemplary embodiment, when it is determined that a local networkconnection is established between the mobile device and the externalservice, controlling the external service with the input commandunderstood by the external service may include transmitting the inputcommand understood by the external service to the mobile device, andcontrolling the external service from the mobile device by transmittingthe input command understood by the external service from the mobiledevice to the external service.

In an exemplary embodiment, when it is determined that a local networkconnection is not established between the mobile device and the externalservice, controlling the external service with the input commandunderstood by the external service may include controlling the externalservice from the server device by transmitting the input commandunderstood by the external service from the server device to theexternal service.

In an exemplary embodiment, a non-transitory computer-readable mediummay contain computer-readable codes to perform the method.

The foregoing and/or other features and utilities of the present generalinventive concept may also be achieved by providing a method ofcontrolling an external service among a plurality of external services,the method including storing a library of commands in a semanticinformation interchange format language in a storage device, eachcommand corresponding to a respective function to be performed at one ofthe plurality of external services, receiving an input command at amobile device, translating the input command into the semanticinformation interchange format language at the mobile device,transmitting the translated command from the mobile device, receivingthe translated command at a server device, determining at the serverdevice if the translated input command is included in the library ofcommands, and if the translated input command is included in the libraryof commands, controlling the external service at one of the serverdevice and the mobile device according to a proprietary language of theexternal service to perform the associated function.

In an exemplary embodiment, the library of commands may include a set ofcommands associated with a specific user.

In an exemplary embodiment, a non-transitory computer-readable mediummay contain computer-readable codes to perform the method.

The foregoing and/or other features and utilities of the present generalinventive concept may also be achieved by providing a method ofcommunication between services, the method including receiving at aserver device an output data from a first external service expressed ina proprietary language of the first external service, converting thereceived output data to a semantic information interchange formatlanguage at the server device, converting the converted output data fromthe sematic information interchange format language to a proprietarylanguage of a second external service at the server device, andtransmitting the converted output data in the proprietary language ofthe second external device to the second external service as an inputcommand to perform a corresponding function at the second externaldevice.

In an exemplary embodiment, a non-transitory computer-readable mediummay contain computer-readable codes to perform the method.

The foregoing and/or other features and utilities of the present generalinventive concept may also be achieved by providing a method ofcontrolling a plurality of external services within a virtual assistantsystem, the method including translating an input command at a mobiledevice into a semantic information interchange language format that canbe communicated between the plurality of external services, the inputcommand representing a function to be performed at one of the externalservices, transmitting the translated input command from the mobiledevice, receiving the transmitted input command at a server device,determining, at the server device, whether the mobile device or theserver device will perform the function at the external service to becontrolled based on at least one of a location of the mobile device withrespect to the external service to be controlled and a networkcommunication established between the mobile device and the externalservice to be controlled, and attaching proprietary language of theexternal service to be controlled to the received input command toperform the associated function by either the mobile device or theserver device according to the determination result.

In an exemplary embodiment, the proprietary language of thecorresponding external service is attached to the received input commandby embedding the proprietary language therein to enable communicationwith the external service to be controlled by the server device or to betransmitted back to the mobile device to directly control the externalservice.

The foregoing and/or other features and utilities of the present generalinventive concept may also be achieved by providing a method ofcontrolling an external service from among a plurality of externalservices connected to a virtual assistant system, the method includingreceiving an input command at a mobile device, the input commandcorresponding to a function to be performed at the external service,translating the received input command into a semantic informationinterchange format language at the mobile device, transmitting thetranslated input command from the mobile device to a server device,determining a location of the mobile device at the server device,determining the function to be performed at the external service, thedetermination being based on the determined location of the mobiledevice, and performing the determined function at the external serviceby converting the translated input command at one of the mobile deviceand the server device into a command recognizable by the externalservice to perform the determined function.

In an exemplary embodiment, the method may further include determiningat the server device whether to convert the translated input command atthe mobile device or the server device based on the determined locationof the mobile device.

In an exemplary embodiment, if the determined location of the mobiledevice is within a predetermined distance of the external service,determining at the server device to convert the translated input commandat the mobile device, otherwise determining at the server device toconvert the translated input command at the server device.

In an exemplary embodiment, the converting the translated input commandincludes determining if a user of the mobile device is authorized toinput commands to the external service, based on at least one of thedetermined location of the mobile device and whether there arecredentials corresponding to the user stored at the server deviceauthorizing the user to input commands to the external service.

In an exemplary embodiment, if the determined location of the mobiledevice is within a predetermined geofenced area, the method may furtherinclude determining that the user of the mobile device is authorized toinput commands to the external service.

In an exemplary embodiment, the method may further include receiving aninput command at the mobile device to control the server device todetermine the location of another mobile device, and to authorize theother mobile device to input commands to the external service while theother mobile device is within a preset area.

In an exemplary embodiment, a non-transitory computer-readable mediummay contain computer-readable codes to perform the method.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other features and utilities of the present generalinventive concept will become apparent and more readily appreciated fromthe following description of the embodiments, taken in conjunction withthe accompanying drawings of which:

FIG. 1A is a block diagram illustrating a virtual assistant systemaccording to an exemplary embodiment of the present general inventiveconcept;

FIG. 1B is a block diagram illustrating a virtual assistant systemaccording to another exemplary embodiment of the present generalinventive concept;

FIG. 2A is a block diagram illustrating an exemplary embodiment of amobile electronic device usable with the virtual assistant systemsillustrated in FIGS. 1A-1B;

FIG. 2B is a block diagram illustrating another exemplary embodiment ofa mobile electronic device;

FIG. 3A is a block diagram illustrating an exemplary embodiment of thesensor part of the mobile electronic device of FIG. 2A;

FIG. 3B is a block diagram illustrating an exemplary embodiment ofexternal input sensors and devices usable with the mobile electronicdevice of FIGS. 2A and 2B.

FIG. 4 is a block diagram illustrating an exemplary embodiment of anassistant server usable with virtual assistant systems illustrated inFIGS. 1A-1B;

FIGS. 5A and 5B are flowcharts illustrating command processes of thevirtual assistant system, according to exemplary embodiments of thepresent general inventive concept;

FIG. 6 is a block diagram illustrating the flow of operations of thevirtual assistant system, according to an exemplary embodiment of thepresent general inventive concept;

FIG. 7 illustrates a process of adaptive addition of a conscious inputcommand to a library of commands, according to an exemplary embodimentof the present general inventive concept;

FIG. 8 illustrates a process of adaptive additions of unconscious inputcommands to a library of commands, according to an exemplary embodimentof the present general inventive concept;

FIG. 9 illustrates a process of consolidating a plurality of gestureinputs in a gesture library to a single gesture input, according to anexemplary embodiment of the present general inventive concept;

FIG. 10 is a diagram illustrating conditional access to the virtualassistant system, according to an exemplary embodiment of the presentgeneral inventive concept;

FIG. 11 is a diagram illustrating access privilege sharing of thevirtual assistant system, according to an exemplary embodiment of thepresent general inventive concept;

FIG. 12 is a diagram illustrating authority sharing and revocation ofthe virtual assistant system, according to an exemplary embodiment ofthe present general inventive concept; and

FIG. 13 is a diagram illustrating mass-addition of services to thevirtual assistant system, according to an exemplary embodiment of thepresent general inventive concept.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to the embodiments of the presentgeneral inventive concept, examples of which are illustrated in theaccompanying drawings, wherein like reference numerals refer to the likeelements throughout. The embodiments are described below in order toexplain the present general inventive concept while referring to thefigures. Also, while describing the present general inventive concept,detailed descriptions about related well-known functions orconfigurations that may diminish the clarity of the points of thepresent general inventive concept are omitted.

It will be understood that although the terms “first” and “second” areused herein to describe various elements, these elements should not belimited by these terms. These terms are only used to distinguish oneelement from another element. Thus, a first element could be termed asecond element, and similarly, a second element may be termed a firstelement without departing from the teachings of this disclosure.

Expressions such as “at least one of,” when preceding a list ofelements, modify the entire list of elements and do not modify theindividual elements of the list.

All terms including descriptive or technical terms which are used hereinshould be construed as having meanings that are obvious to one ofordinary skill in the art. However, the terms may have differentmeanings according to an intention of one of ordinary skill in the art,case precedents, or the appearance of new technologies. Also, some termsmay be arbitrarily selected by the applicant, and in this case, themeaning of the selected terms will be described in detail in thedetailed description of the invention. Thus, the terms used herein haveto be defined based on the meaning of the terms together with thedescription throughout the specification.

Also, when a part “includes” or “comprises” an element, unless there isa particular description contrary thereto, the part can further includeother elements, not excluding the other elements. In the followingdescription, terms such as “unit” and “module” indicate a unit toprocess at least one function or operation, wherein the unit and theblock may be embodied as hardware or software or embodied by combininghardware and software.

Hereinafter, one or more exemplary embodiments of the present generalinventive concept will be described in detail with reference toaccompanying drawings.

FIG. 1A illustrates a virtual assistant system 10 according to anexemplary embodiment of the present general inventive concept, and FIG.1B illustrates a virtual assistant system 10 according to anotherexemplary embodiment of the present general inventive concept.

The virtual assistant system 10 according to the exemplary embodimentsof FIGS. 1A and 1B may include a mobile electronic device 100, anassistant server 200, and an external device 300 a (or devices 300 a1-n) and/or service 300 b (or services 300 b 1-n).

The mobile electronic device 100 (hereinafter “mobile device”) includescapabilities of wireless or wired connectivity with the Internet andvarious networks in which the mobile device 100 can be configured tocommunicate. More specifically, the mobile device 100 may communicatewith the assistant server 200 and/or the external device(s) 300 a andservice(s) 300 b) using Wi-Fi, Wi-Fi direct, Near-field communication,Bluetooth, radio frequency identification (RFID), Ethernet, FireWire,universal serial bus (USB), high definition multimedia interface (HDMI),or any other type of wireless or wired connection, but is not limitedthereto. The communications between the mobile device 100, the assistantserver 200, and the external device(s) 300 a 1-n and service(s) 300 b1-n may be performed via the Internet, including cloud computingapplications, satellite, a local network, etc.

The mobile device 100 may be any type of computing platform, including asmartphone, a tablet computer, a laptop computer, a smart watch, or awearable device, but is not limited thereto, and may even includenon-mobile devices such as a desktop computer or other stationarycomputing devices. The mobile device 100 is described in greater detailbelow with respect to FIG. 2A.

The mobile device 100 may also include an assistant interface 110 (alsoreferred to as “the Client”) which is specifically created as part ofthe virtual assistant system 10 to allow all actions, commands, etc.,originating at the mobile device 100, or obtained by the mobile device100 via another sensor-type device, to be communicated to the assistantserver 200, and vice versa, to allow the assistant server 200 tocommunicate necessary information to the mobile device 100 in which theassistant interface 110 is provided. The assistant interface 110 alsoallows communications with other assistant interfaces 110 a-n providedon another user's mobile device 100 a-n, or other servers which maycontain libraries of commands, to be described in more detail infra. Theassistant interface 110 provided at the mobile device 100 can performsimilar operations as a “plugin” or “connector,” among other operationsto be described herein. The assistant interface 110 is described ingreater detail below with respect to FIGS. 2A, and 5A-5B.

Alternatively, as illustrated in FIG. 2B, the assistant interface 110may be provided on the same hardware installation as the assistantserver 200. In other words, the assistant server 200 may be provided atthe mobile device 100 together with the assistant interface 110, inwhich case the assistant interface 110 is not required to communicatewith any backend-type server system in order to understand the actionsrequired to take in order to perform any functions at a device 300 a orservice 300 b. Thus, in the exemplary embodiment as illustrated in FIG.2B, to be discussed in more detail below, the virtual assistant system10 may be entirely self-contained in a mobile type device as definedabove.

The assistant server 200 is a server that may include a computer, adedicated computer (i.e., a back-end server), or a plurality ofcomputers, running one or more application(s) (i.e., software) thatallows communication with the mobile device 100 via the Internet, andalso allows communication with an infinite range of external devices 300a and services 300 b (i.e. a television, a mobile device application,etc.) via the Internet. The assistant server 200 described here andillustrated in FIGS. 1A and 1B has an open type platform 210, i.e. afacility whereby any software engineer (also referred to as the softwaredeveloper) may create a connector (a “plugin” 220) therein to correspondwith any device(s) 300 a or service(s) 300 b. The assistant serverplatform 210 and plugins 220 may be included all together in onehardware configuration of the assistant server 200.

It is well known that each of the external devices 300 a and services300 b used throughout the world have their own corresponding proprietarylanguage as well as specific credentials which are required tocontrol/operate the respective external device 300 a or service 300 b.Each plugin 220 allows the assistant server 200 to communicate with therespective external device 300 a or service 300 b in its respectiveproprietary language via the Internet, WiFi, Bluetooth, WiFi direct,HDMI, USB, FireWire, NFC or other means in order to execute a user'swishes, either explicit or inferred by the virtual assistant system 10.The plugin 220 also can provide the assistant server 200 with anyproprietary language information regarding the corresponding externaldevice 300 a or service 300 b, including specific credentials, which maybe necessary in order to permit security access to the desired device orservice to be controlled.

The various plugins 220 a-n correspond to various devices, services,programs, and applications, but are not limited thereto. Anengineer/developer may create such a plugin 220 at the assistant server200 to allow the assistant server 200 to communicate with thecorresponding external device 300 a (or service 300 b) in its respectiveproprietary language. The engineer/developer will create the plugin 220for the corresponding device 300 a or service 300 b based onspecifications of the external device 300 or service 300 b. Accordingly,an infinite number of plugins 220 can be created at the assistant server200 to allow a user to control the various respective devices 300 a 1-nand services 300 b 1-n in the manner described above.

The assistant server 200 also has a designated storage 250 (illustratedin FIG. 4) associated with it. The storage 250 may be included as partof the assistant server 200 (i.e. a back-end server), or it may be anexternal storage which the assistant server 200 can access, such as, forexample, an external server, a remote storage device or a cloud storageservice. The software engineer can select from a predefined library ofcommand inputs which are stored in the designated storage 250 in orderto define a command set 240 (illustrated in FIG. 4) which allows theuser to present corresponding commands to each external device 300 a 1-nand service 300 b 1-n via the assistant server 200. The library ofcommand inputs may include any inputs which may be received by themobile device 100. The command inputs may include gestures and icons, aswell as audible or voice commands (stored as actionable data), and otherforms of commands, which are stored in the storage 250 (to be describedin more detail below with respect to FIG. 4) associated with theassistant server 200, to define a command set 240 which allows the userat the mobile device 100 to present these selected commands to therespective external devices 300 a 1-n and services 300 b 1-n via theassistant server 200. In other words, the storage 250 stores a vastvocabulary of gestures and icons (and voice data) in the form of acommon vocabulary format which can be selected from by the softwareengineer in order to create separate command sets 240 a-n associatedwith an external device 300 a or service 300 b. These commands areselected to create a command set 240 to correspond with respectivefunctions that can be performed at the external devices 300 a andservices 300 b. The commands stored in the storage 250 can also beshared between an infinite number of the external devices 300 a 1-n,services 300 b 1-n and mobile devices 100 a-n.

As will be described below with respect to FIG. 1B, under certainpredefined situations, these specific commands from the defined commandset 240 can be presented to the corresponding external device 300 a orservice 300 b directly from the mobile device 100 through the assistantinterface 110 in a local control mode. The assistant server 200 alsoprovides assistance in this local mode, as will be described in moredetail infra.

As pointed out above, the selections of commands by the engineer can begrouped together and saved in the storage 250 as separate command sets240 a-n, which can be specific to the corresponding device 300 a andservice 300 b (illustrated in FIG. 4). In other words, storage 250 canbe configured with a vast vocabulary of gestures, icons, voice data, andother type commands provided in a common vocabulary format, which can beshared between all devices 300 a and services 300 b which are connectedto the assistant server 200. This common vocabulary format of commandswill be hereinafter referred to as a “semantic information interchangelanguage format,” from which a software engineer may select from whilecreating a plugin 220 in order to create a command set to be associatedwith the various functions or operations that the corresponding externaldevice 300 a and service 300 b can perform.

The semantic information interchange format language used at theassistant sever 200 allows the assistant interface 110, provided withinthe mobile device 100, to communicate the many gestures, icon touches,voice data, etc., from the mobile device 100 (in raw input form) to theassistant server 200 since the assistant interface 110 translates thereceived gestures, icon touches, voice data, etc. into this createdsemantic information interchange format language in order to communicateall types of information relevant to different devices 300 a 1-n andservices 300 b 1-n therebetween via a single language within theassistant server 200.

The semantic information interchange format language allows theassistant server 200 to function as a universal translator between thevarious devices 300 a and services 300 b. More specifically, thesemantic information interchange format language can be used forcommunication between the assistant server platform 210 and each of theplugins 220 for all of the devices 300 a 1-n and services 300 b 1-n forwhich a plugin 220 is created at the assistant server 200. In thismanner, various external devices 300 a 1-n and/or services 300 b 1-n maycommunicate information between each other or with the mobile device 100(and other mobile devices 100 a-n), which will be described in moredetail infra.

The assistant server 200 may include any type of master/slavearrangement as predetermined by a user. This may be predetermined by theend-user, or someone that is managing the assistant server can also setup the arrangement of the assistant server 200 in a predetermined way.The assistant server 200 may also be connected to any number of mobiledevices 100 a-n, and may therefore facilitate communication betweenthese mobile devices 100 a-n, as will be described in greater detailinfra.

As described above, the storage 250 can store various languages, icons,gestures, sensor-commands, key-strokes, voice data commands, programs,applications, etc., as well as libraries including any combinations ofthe above items and commands in the semantic information interchangelanguage format. These items and commands are associated with anycommands that can be instructed by a user by inputting such a command atthe mobile device 100, which then sends the input command to theassistant server 200 via the assistant interface 110 after beingtranslated into the semantic information interchange language format bythe assistant interface 110. Alternatively these items or commands canbe obtained by the mobile device 100 via another source (i.e., sensors160, to be described in more detail below with reference to FIGS. 2A,3A-3B, and 6) and then presented to the assistant server 200 via theassistant interface 110 after being translated into the semanticinformation interchange language format by the assistant interface 110.

As pointed out above, the assistant interface 110 first translates aninput command into the semantic information interchange language format.Once the command is provided to the assistant server 200, the commandmay be converted to the appropriate proprietary language at thecorresponding plugin 220. The proprietary language command provided bythe plugin 220 contains within it specific credentials with respect tothe device 300 a or service 300 b, such as, for example, where thedevice 300 a or service 300 b is located, the IP address of the device300 a or service 300 b, and/or any other information that may berequired to permit the virtual assistant system 10 to directly controlthe desired device 300 a or service 300 b. Thus, each plugin 220 can becreated by an engineer to contain specific credentials of thecorresponding device 300 a or service 300 b required in order tocommunicate with and operate the device 300 a or service 300 b.

In other words, any type of gesture, icon touch, voice, keyboard input,or other input command input at the mobile device 100 or through themobile device 100 (a raw input command) by another sensor type device,can be first translated into the semantic information interchange formatlanguage via the assistant interface 110. Once the semantic informationinterchange language format command is recognized within the assistantserver 200, it can be translated at each plugin 220 created for thecorresponding external device 300 a (or service 300 b) to theproprietary language (including any necessary credentials) of thecorresponding external device 300 a (or service 300 b). Using theproprietary language format (including any necessary credentials) of theinput command, the intended function/operation desired to be performedat the corresponding external device 300 a or service 300 b can beperformed.

As illustrated, for example, in FIG. 1A, the assistant server 200 maycommunicate directly with the external devices 300 a 1-n or services 300b 1-n by converting the received commands in the semantic informationinterchange format language to the corresponding proprietary language atthe plugins 220. In this exemplary embodiment, the mobile device 100 isconfigured to be part of the virtual assistant system 10 bycommunicating with the assistant server 200 through the assistantinterface 110. The assistant server 200 can control (via a plugin 220)any number of the external devices 300 a 1-n and services 300 b 1-nincluded within the virtual assistant system 10 via an Internet or anyother type of wireless connection, as described above, which theassistant server 200 shares with the external devices 300 a and services300 b. Furthermore, the assistant server 200 can provide interaction(share information) between each of any number of the external devices300 a and services 300 b included within the virtual assistant system 10using the semantic information interchange format language. Theseexternal devices 300 a and services 300 b should be connected to theassistant server 200 via an Internet or other type of wirelessconnection, as described above.

Alternatively, the assistant interface 110 may directly performfunctions associated with an input command at the device 300 a orservice 300 b. FIG. 1B is a block diagram illustrating this exemplaryembodiment of the present general inventive concept. As illustratedtherein, the mobile device 100 is configured to be part of the virtualassistant system 10 by communicating with the assistant server 200through the assistant interface 110, similar to the exemplary embodimentof FIG. 1A. However, in this exemplary embodiment, when it is determinedthat the mobile device 100 is within a certain range of a particularexternal device 300 a or service 300 b desired to be controlled, orwithin a common network of the device 300 a or service 300 b, theassistant server 200 can communicate back to the mobile device 100 topermit the mobile device 100 to perform direct control of the externaldevices 300 a and services 300 b, similar to the way a remote controldevice would perform these controls (i.e. a television and correspondingremote control device).

More specifically, if a location based communication service (i.e., NFC,Bluetooth, etc.) is established between the mobile device 100 and adesired external device 300 a or service 300 b to be controlled, this“established proximity” communication information can be provided to theassistant server 200 through the assistant interface 110. The assistantserver 200 can then provide control access of the desired externaldevice 300 a or service 300 b to the mobile device 100 directly. In thissituation, the plugin 220 corresponding to the device 300 a or service300 b can first embed the necessary proprietary language informationinto the semantic information interchange format language so that theassistant server 200 can provide this information to the mobile device100 through the assistant interface 110. Here the semantic informationinterchange format language can be translated into the proprietarylanguage which the mobile device 100 can use to control the device 300 aor service 300 b. As a result, the mobile device 100 can directlycontrol the desired external device 300 a or service 300 b in the samefashion as a hand-held remote control specifically designed for thedevice 300 a or service 300 b would control the device 300 a or service300 b.

A more detailed overview of the control process of the virtual assistantsystem 10 is provided below with reference to FIGS. 5A and 5B.

The external device(s) 300 a or service(s) 300 b may include any type ofdevice or service that is connectable to the mobile device 100 and/orthe assistant server 200 via a local network or the Internet, or otherwireless form of communication, as described above. Examples of devices300 a and services 300 b for which plugins 220 may be created includehome and office systems (i.e., security, lighting, thermostats),automotive devices, apps, wearable devices, online services, etc. Thislist of devices 300 a and services 300 b are provided as examples only,and are far more expansive to millions of other devices and services.

Services 300 b, such as, for example, “apps,” usually found on smartphones, are accessible via the Internet and the mobile devices 100 a-nthemselves, and devices 300 a, including, but not limited to, homeappliances, lights, etc. are also capable of being Internet-accessibleby being supplied with a wireless chip or other Internet-accessibledevice. This concept has been recognized and is currently being referredto in discussions relating to “the internet of things,” also referred toas IoT. Such devices 300 a are also accessible via a local network typeof communication so that the mobile device 100 may ultimately controlthe device 300 a directly once the proprietary language includingnecessary credentials is obtained by the mobile device 100.

It is to be noted that an infinite number of external devices 300 a 1-nand services 300 b 1-n may be accessed and controlled by the virtualassistant system 10, and any number of mobile devices 100 a-n can beincluded into the virtual assistant system 10 to access and control theinfinite number of external devices 300 a and services 300 b. Access tothe devices 300 a and service 300 b may also be shared between mobiledevices 100 (see dotted lines “sharing access” in FIG. 1A), as will bedescribed in more detail infra.

As pointed out above, the semantic information interchange formatlanguage created within the assistant server 200 acts as a universalinterchange format type language to allow communication and exchange ofinformation between external devices 300 a and services 300 b throughthe assistant server 200 via respective plugins, between the mobiledevice 100 and the assistant server 200, between the mobile device 100and any of the infinite number of external devices 300 a and services300 b, and between mobile devices 100 a-n. An example of a list of inputcommands can include the following: a voice command can be captured by amicrophone connected to or part of a mobile device (such as a smartphoneor tablet) or other computing hardware. Gestures can be captured by acamera, motion detector, proximity detector, infrared sensor,temperature sensor, global positioning device, or any otherphoto-detecting device that is capable of capturing information about auser's movements, which is in turn connected to a mobile or othercomputing device. Gestures may also be captured by wearable, held, orother devices that detect movement by using accelerometer hardware orother movement-tracking hardware, or even eye-tracking, lip-tracking orfacial-tracking systems. Icons can be touched or clicked either by handor a stylus on a touch sensitive service (such as that of a mobiledevice or other touchscreen) or by means of a click by positioning amouse or roller-ball on top of the icon. In addition, it is possiblethat the user may use voice alone, gestures alone or icons alone, or anycombinations thereof to make the assistant do these tasks. For example,a user may gesture upwards at a camera to which the assistant has accessto indicate that a thermostat controlled by the assistant should raisethe temperature. Alternatively, for example, this gesture can bedetected by means of an armband or other wearable device.

In creating a command set for a plugin 220, an engineer/developer mayselect from gestures that are part of the predefined semanticinformation interchange format language specified within the assistantserver system 10. For example, if one closed first and two closed fistsare part of the predefined semantic information interchange formatlanguage stored in the virtual assistant system 10, anengineer/developer may select a gesture of closing a first to representthe locking of a door and closing two fists to represent engaging adeadbolt.

As another example, a user may hold a picture up to a camera (associatedwith the user's mobile device 100) and with a waving gesture instructthe assistant server 200 to identify a person in the picture via aface-recognition program (a service, i.e., an “app”) to which a plugin220 has been created. This specific gesture of waving a picture togetherwith using such a face-recognition program will have been previouslyprovided within the semantic information interchange format language ofgestures stored in the storage 250. Then this gesture may be selected byan engineer while creating a plugin for a face recognition type device300 a or service 300 b.

Because the library of commands programmed at the assistant server 200in the semantic information interchange format is constant across alimitless (thousands or even millions) number of plugins, similarcommands emerge for similar concepts. As an example, two separateplugins 220 could be respectively created for a door and a billboard,respectively, in a public square, and stored within the assistant server200. If, for example, closing a first is associated with the concept ofshutting down or closing up for the day, a user may then utilize a samegesture of closing a first to signify locking the door as well asshutting down the billboard, depending on whether the user wishes tocontrol the door or the billboard.

Since plugins 220 are created for both the door and the billboard (aswell as an infinite number of other devices, services, programs,applications, etc.), both the door and the billboard are able tounderstand the commands initiated by the user through the mobile device100, as the assistant server 200 acts as a translator or an interpreter(via the semantic information interchange format language created in theassistant server 200, based on the library of gestures, icons, and othercommand inputs stored therein) between the mobile device 100 and thedoor and the billboard. More specifically, the assistant server 200 actsas the translator between the infinite number of external devices 300 a,services 300 b, etc., and the mobile device 100 by using the semanticinformation interchange format language. As pointed out above, thesemantic information interchange format language (the common vocabularybetween all devices, services, etc.) is specifically created within theassistant server 200 to allow communication between the assistant serverplatform 210 (see FIG. 1A) and each of the plugins 220 a-n for all ofthe devices 300 a 1-n and services 300 b 1-n in which a plugin 220 iscreated at the assistant server 200, and between the assistant serverplatform 210 and the assistant interface 110.

FIG. 2A is a block diagram illustrating an exemplary embodiment of amobile device 100 in detail for purposes of describing how the mobiledevice 100 is configured to be a part of the virtual assistant system 10illustrated in FIGS. 1A and 1B.

The mobile device 100 may include the assistant interface 110, acommunicator 105, a controller 120, a central processing unit (CPU) 130,a storage unit 140, a user interface 150, a sensor 160, at least onespeaker 170, at least one microphone 180, and at least one camera 190.

Referring to FIG. 2A, the communicator 105 can be used to establish aconnection with the assistant server 200 and/or an external device 300 aor service 300 b. The communicator 105 may include an antenna or anytype of communication device to establish the connection with theassistant server 200 and the external device 300 a using Wi-Fi, Wi-Fidirect, NFC, Bluetooth, RFID, Ethernet, FireWire, universal serial bus(USB), high definition multimedia interface (HDMI), or any other type ofwireless or wired communication method, but is not limited thereto. Thecommunications between the mobile device 100, the assistant server 200,and any external device(s) 300 a and service(s) 300 b may be performedvia the Internet, including cloud computing applications, satellite,etc.

The controller 120 may control communications between the mobile device100 and the assistant server 200, as well as control communicationbetween the mobile device 100 and the external devices 300 a or services300 b under certain conditions being met, such as a location basedcommunication service (i.e., near field communication (NFC), Bluetooth,etc.) being established, as described above with respect to FIG. 1B.

The controller 120 determines which device the assistant interface 110is communicating with in order to interpret a received request whetherthe request is received in a proprietary language format or in thesemantic information interchange language format. Although the devicewhich the assistant interface 110 is communicating with is often theassistant server 200, the libraries of possible inputs (icons, gestures,voice commands, with interpretations) could be obtained from a differentserver or from another assistant interface 110 a-n on a separate mobiledevice 100 a-n. For example, multiple assistant interfaces 110 a-n couldwork collaboratively: i.e., once one assistant interface 110 a haslearned how to interpret an input, this assistant interface 110 a couldshare the input(s) with another assistant interface. Accordingly, thecontroller 120 determines what device the assistant interface 110 istalking to in order to provide these features.

The controller 120 can thus make a determination that the assistantinterface 110 should reach out to an external source other than theassistant server 200 under certain circumstances. For example, if theassistant interface 110 is determined by the controller 120 to bereceiving an input request that requires information from anotherassistant server 200 a-n or another assistant interface 110 a-n, thecontroller 120 may dictate to the assistant interface 110 to reach outto that external other source. The assistant interface 110 wouldaccordingly connect to the external other source by way of thecommunicator 105 and present the request. The external other source willcheck the full set of libraries to interpret this request (gesture, icontouch, voice data, etc.), and if found, the external other source willrespond to the assistant interface 110 and tell it what to do in thesemantic information interchange language format (with proprietarylanguage embedded therein). At this point if the mobile device 100 isoperating through the assistant server 200 as illustrated in FIG. 1A,the assistant interface 110 will provide the command to the assistantserver 200 to perform the command on the intended external device 300 aor service 300 b. Alternatively, if the mobile device 100 is operatingin a local control mode as illustrated in FIG. 1B, the assistantinterface 110 can then connect with the device 300 a through thecommunicator 105 and directly control the intended external device 300 aor service 300 b using the properly obtained proprietary language.

The CPU 130 may request the virtual assistant system 10 to runapplications and/or programs, perform various functions of the mobiledevice 100, and adaptively learn various commands based on the user'spreferences, among various other uses. The CPU 130 may also run variousoperating systems that may be changed, altered, or personalized, basedon the user's preferences. Further, when there are a plurality ofexternal devices 300 a 1-n and/or services 300 b 1-n in which the userdesires to control, the CPU 130 may perform various functionsspecifically corresponding to the plurality of external devices 300 aand/or services 300 b, based on respective user commands.

The storage unit 140 may store the commands learned by the CPU 130, aswell as the various languages, icons, gestures, sensor-commands,programs, applications, commands, and libraries including anycombinations of the above items, that are downloadable by the mobiledevice 100 from the assistant server 200. For example, if the mobiledevice 100 does not possess a specific command among the library ofcommands within the assistant server 200 to control a particular device300 a or service 300 b, the mobile device 100 can download any singlecommand or set of commands from the assistant server 200, at which pointthe mobile device 100 can directly control the device 300 a or service300 b, or at least have a more extensive library of commands tounderstand a command being input to the mobile device 100. In otherwords, if a user of the mobile device 100 makes a gesture or provides avoice command which is not a readily known command to the mobile device100, the assistant server 200 can recognize the correct command bychecking within the extensive library of commands, while consideringvarious contexts of the command and the user information collected viathe history of information previously gathered, as described above.Then, with the help of a corresponding plugin 220, the assistant server200 can generate the command in the semantic information interchangelanguage format with the proprietary language embedded therein. Thecommand can then be provided to the mobile device 100, at which pointthe mobile device 100 can download the correct command into the storageunit 140, translate the command to the proprietary language of thedevice 300 a or service 300 b via the assistant interface 110, and thenapply the perform an intended function at the device 300 a through thecommunicator 105 by using the proprietary language of that device 300 a.

The user interface 150 allows a user to perform various functions on themobile device 100, including displaying selectable icons, makingtelephone calls, accessing the Internet to view various web pages,playing multimedia such as music and movies, viewing pictures, runningprograms, controlling an external device 300 a or service 300 b,accessing the assistant server 200, modifying the mobile device 100,etc., but is not limited thereto. The user interface 150 may include ascreen, a keyboard, a keypad, a mouse, a trackball, a touch screen, anaudio interface (for example with the at least one microphone 180), ahologram interface, a sensing interface, etc., but is not limitedthereto. Moreover, the user interface 150 may include any combinationsof the various types of aforementioned user interfaces.

The sensor 160 may be configured to sense a sound, gesture, or touch ofthe user. The sensor 160 may work in conjunction with facial recognitionsoftware running on the CPU 130 to allow the mobile device 100 torecognize facial expressions of the user, or a ring type device, suchas, for example, Fin™, which is a Bluetooth enabled device that canprovide commands from gestures made by a user. In this example, thecommands can be received by the CPU 130 and translated into the sematicinformation interchange language format by the assistant interface 110.Additionally, the sensor 160 may include other types of sensing devicesthat sense movement of the mobile device 100 itself, such as a gyration,pressure, or proximity of the mobile device 100 to a device 300 a to becontrolled. Moreover, the sensor 160 may include global positioningsystem (GPS) technology to sense a location of the mobile device 100.The sensor 160 may include a single sensor having multiple functions, ormay be divided into a plurality of different sensors based on the user'spreference. The sensor 160 is described in greater detail below withreference to FIG. 3A.

The mobile device 100 may further include the at least one speaker 170to allow the user to hear audio emitted from the mobile device 100, theat least one microphone 180 to allow the user to speak into the mobiledevice 100, and the at least one camera 190 to allow the user to captureimages. The at least one speaker 170 and the at least one microphone 180may be used to perform telephone operations, and may also be used toreceive audio commands (for example voice commands) from the user thatmay be processed by the CPU 130. The at least one camera 190 may be usedto photograph still or moving objects, to perform webcam communicationswith other users using applications such as Skype or GChat, or toreceive/read gestures and/or facial features from the user that may beprocessed by the CPU 130.

The assistant interface 110 (also referred to as “Client”), as describedabove with respect to FIGS. 1A-1B, can perform the operations of aplugin by translating input commands into the semantic informationinterchange language format and can communicate with the assistantserver 200 using the semantic information interchange format language.Furthermore, the assistant interface 110 can translate any of thecommands received at the mobile device 100 by any form of the sensor 160thereof or received by the mobile device 100 by way of an externalsensor, such as, for example only, the wearable ring as described above.The assistant interface 110 also translates data received from theassistant server 200 into a format which may be understood by the mobiledevice 100, and which may be displayed for example on the user interface150.

As pointed out above, FIG. 2B illustrates an exemplary embodiment wherethe mobile device 100 of FIG. 2A may include the assistant server 200therein as well as the other devices illustrated in FIG. 2A. In thiscase, the assistant interface 110 is not required to communicate withany backend-type server system in order to understand the actionsrequired to take in order to perform any functions at a device 300 a orservice 300 b when operating in a local mode. Thus, in this exemplaryembodiment illustrated in FIG. 2B, the virtual assistant system 10 maybe entirely self-contained in a mobile type device 100, as describedabove. In the embodiments described above with reference to FIGS. 1A-1Band 2A, the interaction between the assistant server 200 and theassistant interface 110 would normally occur over the web. However, inthis exemplary embodiment illustrated in FIG. 2B, the interactionbetween the assistant server 200 and the assistant interface 110 occurswithin the mobile device 100 itself. Furthermore, the storage 250associated with the assistant server 200 is also provided on the samehardware as the mobile device 100, and is used by the assistant server200 similar to the way the assistant server 200 uses the storage 250 asdescribed in FIGS. 1A-1B. Thus the storage 250 in FIG. 2B isfunctionally equivalent to the storage 250 described in FIGS. 1A-1B.Although many of the components of the mobile device 100 of FIG. 2A arenot illustrated in the mobile device 100 of FIG. 2B, such as thecommunicator 105, the CPU 130, the controller 120, the U.I. 150, thesensor 160, the storage unit 140, etc., in order to highlight thedifferences between the mobile device of FIG. 2A and FIG. 2B, thesecomponents are also included in the mobile device of FIG. 2B.

A virtual assistant system 10 according to this exemplary embodiment ofFIG. 2B is valuable in cases where communication is poor, such as remoteapplications, military applications, etc. Since the mobile device 100and the assistant server 200 of this exemplary embodiment do not rely onwireless communications between each other to perform functions atdevices 300 a or services 300 b, the probability of failure incontrolling an intended device 300 a or service 300 b as a result ofpoor wireless communication services can be eliminated. Furthermore,users who do not want their data to be stored at some remote locationmay also use this type of setup of the virtual assistant system 10 forsecurity purposes, or other similar reasons. Since no assistantinformation is being sent wirelessly, it is more secure. This type ofsetup can also be very valuable in automobiles or aircraft which facevarying connectivity situations, i.e., sometimes becoming disconnectedfrom wireless service.

FIG. 3A is a block diagram illustrating an exemplary embodiment of thesensor 160 of the mobile device 100 illustrated in FIG. 2A. The sensor160 as illustrated in FIG. 3A can be provided in the form of a pluralityof different sensors.

The sensor 160 may include various different sensors located at variouslocations on the mobile device 100, and may include a touch screen 161,a pressure sensor 162, a motion sensor 163, a heat sensor 164, aproximity sensor 165, a gyration sensor 166, and a global positioningservice (GPS) sensor 167.

The touch screen 161 is a display that displays items, icons, and mediathereon, while allowing the mobile device 100 to detect a location ofthe user's finger, stylus, etc., when in contact with the touch screen161. As such, when the user moves the finger around the touch pad,various commands can be performed on the touch screen 161, such asscrolling through menu items, selecting icons, flipping pages, etc., butis not limited thereto.

The pressure sensor 162 allows the mobile device to detect a pressure atwhich the touch screen 161 is depressed. As such, different pressuressensed by the pressure sensor 162 can result in different commandsexecuted by the mobile device 100. The pressure sensor 162 may be acomponent that is separate from the touch screen 161, or may beintegrated therein.

The motion sensor 163 can detect a motion of the user or a motion of themobile device 100 with respect to the user or the external device 300 a.As such, the motion sensor 163 can actively track the user's movement.The motion sensor 163 may be a component that is separate from the atleast one camera 190, or may be integrated therein.

The heat sensor 164 can sense heat from an outside source, which can beprocessed and interpreted as data by the CPU 130.

The proximity sensor 165 can sense a proximity of the mobile device 100with respect to other objects and/or users. The proximity sensor 165 maybe a component that is separate from the at least one camera 190, or maybe integrated therein.

The gyration sensor 166 can sense a motion and/or a direction of motionof the mobile device 100 itself in a three-dimensional space. Morespecifically, the gyration sensor 166 can sense how the mobile device100 is moving, thereby allowing the mobile device 100 to be used as awand-like apparatus to control the external device 300 a or service 300b, for example. For instance, the gyration sensor 166 may sense that theuser moves the mobile device 100 from a low position to a high position,and then may send a command through the assistant server 200 to controla television (corresponding to an external device 300 a) to increase itsvolume. Alternatively, the motion sensor 163, the proximity sensor 165,and the at least one camera 190 may also be utilized to sense that themobile device 100 has moved from a low position to a high position andsend a corresponding command to the assistant server 200.

The GPS sensor 167 can sense a location of the mobile device 100 usingGPS technology, in order to substantiate an exact location of the mobiledevice 100. When the location of the mobile device 100 is substantiated,the assistant server 200 can determine which external devices 300 a andservices 300 b can and/or should be controllable by the mobile device100, and which types of commands can and/or should be utilized toperform the controls.

According to an exemplary embodiment, if a user is operating the mobiledevice 100 outside his/her house and wants to open a garage door (adevice 300 a) which is wirelessly connected to the assistant server 200,the user may give a “thumbs up” hand sign to the at least one camera190, or using another form of sensor, at which point the assistantinterface 110 translates and provides the command to the assistantserver 200 to communicate the command to the garage door plugin, causingthe garage door to open. The “thumbs up” hand sign is interpreted tosignify “garage door open” at this point because the GPS sensor 167 cansense that the mobile device 100 is outside the user's house. However,if the user is inside the house, the same “thumbs up” hand sign can beused to close the garage door at this point because the GPS sensor 167can sense that the user is inside the house. Therefore, the same handsign or other gesture can be used to perform different functionscorresponding to a single external device 300 a, based on a location ofthe mobile device 100 sensed by the GPS sensor 167.

As another exemplary embodiment, a sprinkler system service 300 b couldbe controlled differently based on a location of the mobile device 100.More specifically, if the GPS sensor 167 senses that the mobile device100 is near the user's home and the user touches a sprinkler system icondisplayed on the user interface (UI) 150 of the mobile device 100, thesprinkler system can be shut off to allow the user to walk on the lawnwithout getting wet. If the user is far from the home and the usertouches the same sprinkler system icon displayed on the UI 150 of themobile device 100, the sprinkler system may turn on, causing water tospray onto the lawn. Alternatively, the user could preset an automatedsprinkler system function based on the GPS sensor 167 sensing a locationof the user.

It will be understood that the sensor 160 may include any combination ofthe sensing devices illustrated in FIG. 3A, or additional sensingdevices not illustrated, according to the particular exemplaryembodiment of the present general inventive concept. For example,gestures can be detected by the at least one camera 190, as well as aphoto-detection device, a wearable device such as an armband or Fin™, orusing an accelerometer or other movement detection device in the mobiledevice 100. In the exemplary embodiment of the present general inventiveconcept illustrated in FIG. 3B, the mobile device 100 is connected to aplurality of external input sensors and devices. For example, the mobiledevice 100 may be connected to external sensors in a user's home,office, or car in order to receive data therefrom which it would not beable to receive with its own sensor 160. External sensing devices, suchas a keyboard, camera, GPS, wearable sensor, microphone, and/or infraredsensor would connect the mobile device 100 to alternate sensing devicesto accept a user input. In this way, the functionality of the user'smobile device 100 can be greatly expanded.

Furthermore, the mobile device 100 may receive input commands fromexternal devices, such as one or more displays, speakers, and actuators.The inputs from these external devices can be processed by the CPU 130to determine such a command being received from the respective externaldevice.

As described above, gestures may also be captured by wearable, held, orother devices that detect movement by using accelerometer hardware orother movement tracking hardware, or even eye-tracking, lip-tracking orfacial-tracking systems. Icons are usually touched or clicked either byhand or a stylus on a touch sensitive service (such as that of the userinterface 150 of the mobile device 100 illustrated in FIG. 2A, or someother form of a touchscreen) or by means of a click by positioning amouse or roller-ball on top of the icon. It is possible that the usermay use voice alone, gestures alone or icons alone, or any combinationsthereof to make the assistant server 200 perform these tasks at thedevice 300 a or service 300 b (i.e., the above example of thecombination of using a GPS together with a sensor to sense the“thumbs-up gesture”). As another example, a user may gesture upwards ata camera to which the assistant server 200 has access to indicate that athermostat controlled by the assistant server 200 should raise thetemperature. Or this gesture can be detected by means of an armband orother wearable device.

FIG. 4 is a block diagram illustrating an exemplary embodiment of anassistant server 200 in detail for purposes of describing how theassistant server 200 is configured to be a part of the virtual assistantsystem 10 illustrated in FIGS. 1A and 1B.

The assistant server 200 illustrated in FIG. 4 may include the assistantserver platform 210 and the plugins 220 a-n.

The assistant server platform 210 may be any device or combinationthereof that allows a software developer to create the plugins 220 atthe assistant server 200. The assistant server platform 210 may, forexample, be a CPU or other computing device used to process commands aswell as communicate with devices and services via the semanticinformation interchange format language. Similarly to the CPU 130 in themobile device 100, the assistant server platform 210 can be continuouslyrunning a learning algorithm or algorithms.

Although plugins 220 a-n are illustrated in FIG. 4 as being separatefrom the assistant server 200, this illustration is provided merely toillustrate the infinite number of plugins 220 that may be created andtheir association with a specific device 300 a or service 300 b. Theseplugins 220 are generally created at the assistant server 200.

As discussed above, the storage 250 may be provided on the same hardwareor cloud service as the assistant server 200, or may be separatelylocated and accessible by the assistant server 200, as illustrated inFIG. 4. The storage 250 may be, for example, a hard drive included inthe assistant server 200, or an external storage device which isconnected to the assistant server 200, for example a flash drive or acloud storage server. The storage 250 may also be a plurality of storagedevices connected to the assistant server 200. The storage 250 storesthe library of commands 230 used to control the devices 300 a andservices 300 b.

The library 230 may be divided into multiple divisions for differenttypes of inputs. For example, the library 230 may be divided into alibrary of icons, a library of voice/word inputs, and a library ofgestures.

The storage 250 may also include separate command sets 240 a-ncorresponding to devices 300 a 1-n and services 300 b 1-n. Each of thenumerous command sets 240 a-n includes the specific commands (associatedwith inputs) which may be used to control a particular one of devices300 a 1-n or services 300 b 1-n.

In order to control a particular device 300 a or service 300 b, the usermay connect the mobile device 100 to the assistant server 200 anddownload the appropriate command set 240. This connection may beachieved via the Internet by using a Wi-fi, Wi-fi direct, NFC,Bluetooth, RFID, or any other type of wireless or wired connection. Thedownloaded command sets 240 may be saved in the storage unit 140, eitherautomatically or after a user acceptance command. Once downloaded, thecommand sets 240 may be used by the assistant interface 110 to translatereceived inputs into the semantic information interchange formatlanguage.

The storage 250 may also store usage data on the virtual assistantsystem 10, including context data (such as a time or a user's physicallocation when certain commands are entered), which inputs are givenincorrectly (which inputs are immediately undone, etc.), and so on. Thisusage data may be made available to plugin developers so that thedevelopers may then update the library 230 accordingly to reflect thisusage data.

The storage 250 may also store updates to the library 230 generatedthrough use of the virtual assistant system 10. As noted above anddiscussed in detail infra, the virtual assistant system 10 may defineadditional inputs, such as gestures, depending on user behavior, whichmay be stored in the storage 250.

A process of determining the intention of an input command and carryingout the corresponding operation at an external device 300 a or service300 b will be described while referring to FIG. 5A. Referring to FIG.5A, when a command is input at the mobile device 100 directly throughthe sensor 160 or via an external sensor (operation S500), the assistantinterface 110 first translates the command input at the mobile device100 into the semantic information interchange language format so thatthe command can be shared among devices 300 a 1-n and services 300 b 1-nconnected to the assistant server 200. As described above, the commandinput at the mobile device 100 may be an icon touch, gesture, voicedata, or any other input provided at the mobile device 100 in which asensor part can receive the input.

The assistant server 200 receives this translated command (operationS510). The connection between the mobile device 100 and the assistantserver 200 may be automatic, or alternatively, the user may be promptedon the mobile device 100 to initiate and/or accept the connection to theassistant server 200.

The assistant server 200 then determines if the virtual assistant system10 is operating in a remote mode or a local mode (operation S520). Thedetermination of whether the virtual assistant system 10 operates inremote mode or local mode can include, for example, whether theassistant server 200 is permitted within the firewall of the device 300a or service 300 b or if the assistant interface 110 is connected withina local network (such as Bluetooth or NFC) to the device 300 a orservice 300 b.

If the assistant server 200 determines that the virtual assistant system10 is operating in remote mode (REMOTE at operation S520), the assistantserver 200 can confirm recognition of the command(s) originating at themobile device 100 by checking among the semantic information interchangelanguage format library of commands at operation S530 a.

If the received command is recognized at the assistant server 200 (YESat operation S530 a), the assistant server 200 can then perform thedesired command instructed (operation S540 a) at the intended externaldevice 300 a or service 300 b via the corresponding plugin 220 createdat the assistant server 200, when the virtual assistant system 10 is inthe remote mode of operation.

Alternatively, if it is determined that there exists any confusion as towhat the command received from the mobile device 100 is intended torefer to (NO at operation S530 a), for example, there may be more thanone device 300 a or service 300 b which uses the same command, theassistant server 200 can examine the command in a number of contexts byconsidering, among other things, the historical usage pattern of thiscommand, the particular user providing this command, the particularlocation in which the command is given, the specific time of day inwhich the command is given, whether this particular user providing thecommand always uses this specific gesture to perform a specific functionat a specific device 300 a or service 300 b, etc., at operation S560 a.In other words, the assistant server 200 will examine all accessiblecontextual information available within a program running at theplatform 210 of the assistant server 200 regarding this received commandin order to determine specifically what possible options could have beenintended by the received command.

Once the assistant server 200 determines which possible options couldhave been intended by the command input at the mobile device 100, theassistant server 200 provides these options of intended commands back tothe mobile device 100 (operation S570 a) so that the options can bedisplayed at the user interface (UI) 150 at the mobile device 100. Herethe user can once again input the intended command, but this time withspecific accuracy by selecting one among a few displayed options. As anexample, a list of icons could be displayed on the mobile devicerepresenting the possible commands, and the user could select one iconas the intended command. The assistant server 200 may also record theuser's selection so that the original command input in operation S500will be recognized if it is input again later (a collection of historydata to form contexts). This is discussed in greater detail below withreference to FIG. 7.

The specifically intended command is then translated at the assistantinterface 110 (into the sematic information interchange language format)and provided to the assistant server 200, where the command is receivedat the assistant server 200 (operation S580 a) to carry out this commandin the remote mode of operation of the virtual assistant system 10. Inother words, when the virtual assistant system 10 is in remote mode, atoperation S540 a, the assistant server 200 may present the command tothe corresponding plugin 220, which converts the command from thesemantic information interchange format language to the proprietarylanguage in which the external device 300 a 1 or service 300 b 1understands. The assistant server 200 then can perform the intendedcommand, i.e., the assistant server 200 can communicate directly withthe external device 300 a 1 or service 300 b 1 in the respectiveproprietary language of the device or service to perform any functionsor operations instructed at the mobile device 100 (or other type sensordevice which is connected to the virtual assistant system 10 and canprovide a command input, which will be described in more detail infra).

Information received from the device 300 a or service 300 b as part ofcarrying out the command in operation S540 a is converted from theproprietary language to the semantic information interchange formatlanguage at the corresponding plugin 220. If additional operations arerequested at other devices or services using this information (YES atoperation S550 a), additional commands input into the mobile device 100are received at operation S510 to perform the additional commands.

Alternatively, when it is determined that the virtual assistant system10 is in a “local mode,” (LOCAL at operation S520), the virtualassistant system 10 can perform the following sequence of operations asillustrated in FIG. 5B.

If the received command is recognized at the assistant server 200 (YESat operation S530 b), the assistant server 200 can then obtain theproprietary language information of the specific device 300 a or service300 b to be controlled, and provide the information to the assistantinterface 110 of the mobile device 100 with the proprietary languageincluded (operation S535 b). With the necessary proprietary languagerequired to perform the intended control obtained at the mobile device100, control of the device 300 a or service 300 b can be directlyperformed at the mobile device 100 through the assistant interface 110(and communicator 105) at operation S540 b. Furthermore, if additionaloperations are requested to be performed at other devices or servicesusing the information obtained as a result of the function performed atthe previously intended device 300 a or service 300 b (YES at operationS550 b), additional commands input at the mobile device 100 are receivedat operation S510 (FIG. 5A) to perform additional functionscorresponding to the additional input commands. These additionalcommands input can include information obtained from the previousoperations performed. For example, continuous communications betweenservices such as “apps” often require information obtained from theprevious “app” in which a function was performed (i.e., obtaininginformation on a movie playing in a user's locale).

If there exists confusion as to what input command may have beenintended when the command is received (NO at operation S530 b), theassistant server 200 then determines which possible options could havebeen intended by the command input at the mobile device 100 (operationS560 b). Similarly to the description above with regard to operationS560 a in FIG. 5A, this determination can be performed by examining thecommand in a number of contexts by considering, among other things, thehistorical usage pattern of this command, the particular user providingthis command, the particular location in which the command is given, thespecific time of day in which the command is given, whether thisparticular user providing the command always uses this specific gestureto perform a specific function at a specific device 300 a or service 300b, etc.

Once the possible intended command options are determined at theassistant server 200 at operation S560 b, the assistant server 200 canobtain the proprietary language of these commands with the correspondingplugins 220, and provide these command options back to the mobile device100 (operation S570 b) with their respective proprietary languageinformation so that the options can be displayed at the user interface(UI) 150 of the mobile device 100. Here the actual intended command canbe selected and performed directly at the mobile device 100 (operationS540 b).

Similarly to operation S580 a described above, the assistant server 200may record the command option selected by the user in operation S570 bso that the command will be recognized if it is input again later.

As noted above, FIG. 5A illustrates a remote mode according to anexemplary embodiment of the present general inventive concept. In theremote mode, the assistant interface 110 does not need to be on the samenetwork or internet as the corresponding device 300 a or service 300 b,nor does the assistant interface 110 need to download the proprietarylanguage of the device 300 a or service 300 b. In remote mode, theassistant interface 110 may be required to present credentials such as ausername and password to prove the user has authorized access to thedevice 300 a or service 300 b, but communication usually occurs throughthe assistant server 200.

Conversely, in the local mode illustrated in FIG. 5B, communicationoccurs directly between the mobile device 100 and the device 300 a orservice 300 b. In local mode, the assistant interface 110 does need tobe on the same network as the corresponding device 300 a or service 300b, specifically the corresponding local network (for example, Bluetoothor NFC). However, once the mobile device 100 is on this local network,credentials are not necessary to prove the user has authorized access tothe device 300 a or service 300 b.

In another exemplary embodiment of the present general inventiveconcept, the virtual assistant system 10 may operate in remote modewhile the assistant interface 110 communicates directly with the device300 a or service 300 b. This is advantageous if the device 300 a orservice 300 b monitors the number of access requests and shuts offaccess to IP addresses that are accessing it excessively. For example,if one million (give or take) users sharing one assistant server 200want to access the same device 300 a or service 300 b through theassistant server 200, according to the control process illustrated inFIG. 5A, the device 300 a or service 300 b would register a large numberof access requests coming from one IP address (the assistant server200). The device 300 a or service 300 b may accordingly block accessfrom the assistant server 200. In this situation, it would be preferablefor each user to access the device 300 a or service 300 b through theassistant interfaces 110 a-n of their respective mobile devices 100 a-n.The device 300 a or service 300 b would then register one access requestfrom each of a million different IP addresses, and therefore would notblock access to any of the users.

The above exemplary embodiment of the present general inventive conceptis illustrated in FIG. 6. In this exemplary embodiment, the virtualassistant system 10 is operating in remote mode. That is, the commandsare being sent to a device 300 a or service 300 b that is not connectedlocally (through Bluetooth, NFC, etc.) to the mobile device 100. In theexemplary embodiment illustrated in FIG. 6, a user input is received atthe mobile device 100 and translated into the semantic informationinterchange format language (operation S610). It is then transmitted tothe assistant server 200 (operations S615), which in turn provides thetranslated input to the corresponding plugin 220 (operation S620). Herethe plugin 220 converts this translated input to the appropriateproprietary language (operation S625).

After the input is converted to the proprietary language, the command issent to the mobile device 100 (specifically to the assistant interface110, through the communicator 105, illustrated in FIG. 2A) (operationS630). The mobile device 100 then transmits the converted command, inthe corresponding proprietary language, to the device 300 a or service300 b (operation S635). According to an exemplary embodiment of thepresent general inventive concept, the converted command may betransmitted by the assistant interface 110 through the communicator 105over the Internet to the device 300 a or service 300 b.

The device 300 a or service 300 b carries out the corresponding actionand transmits data back to the assistant interface 110 (operation S640).This data is in the proprietary language of the device 300 a or service300 b, which the mobile device 100 on its own does not understand. Themobile device 100 accordingly sends the received data to the assistantserver 200 (operation S645), which provides it to the correspondingplugin 220 (operation S650). The plugin 220 converts the data into thesemantic information interchange format language (operation S655).Similarly to the remote mode illustrated in FIG. 5A, once the data isconverted to the semantic information interchange format language, itmay be shared between the mobile device 100 and other devices 300 a andservices 300 b.

Furthermore, the present general inventive concept is not limited toonly using one of the local or remote mode at a time. For example, themobile device 100 could be connected locally to one or more devices 300a and services 300 b, while also being connected through the assistantserver 200 to a different device 300 a or service 300 b.

As described above, each plugin 220 created at the assistant server 200for the specific external device 300 a or service 300 b is configuredsuch that the plugin 220 can convert commands in the semanticinformation interchange format language into the correspondingproprietary language. Similarly, the plugin 220 can also convert thisproprietary language and other information (i.e., credentials of thedevice or service) into the semantic information interchange formatlanguage used within the assistant server 200 so that the differentexternal devices 300 a 1-n and services 300 b 1-n connected to theassistant server 200, via the Internet or other type of wirelesscommunication used, as described above, can now share informationbetween one another.

Furthermore, information obtained at the mobile device 100 from onedevice 300 a or service 300 b, as the result of an input command at themobile device 100, can then be communicated and shared with otherdevices 300 a or services 300 b via the assistant server 200 so that auser of the mobile device 100 can communicate information with andbetween each of the devices 300 a and/or services 300 b. In other words,the mobile device 100 can control each of the devices 300 a and/orservices 300 b and interact (share information) between the devices 300a and/or services 300 b to obtain a greater or more substantive bundleof information, which is all transported in the semantic informationinterchange language format. This newly obtained substantive bundle ofinformation can then be used with another command to be directed toanother device 300 a or service 300 b or shared with other mobiledevices 100 a-n connected with the assistant server 200. This processcan continue between external devices 300 a and services 300 b and themobile device 100 as well as between mobile devices 100 a-n connectedwith the assistant server 200 to share and obtain any informationdesired, and consequently to use this information to access anyadditional external device 300 a or service 300 b for an ultimate resultsought.

An example of gathering information between services 300 b, for example,can be a case where a user of a mobile device 100 accesses a service 300b Fandango™ to determine which movies may be available in a givenlocation, and then choose a specific movie desired to view. Before theuser purchases tickets to this movie, the user can transmit this movieinformation (translated into the semantic information interchange formatlanguage by the assistant interface 110) to another service, such asIMBD™. Then the user can determine which actors are in the movie via theIMDB™ app prior to purchasing the tickets through Fandango™, based onwhether the user may desire to watch a specific movie if the movieincludes this particular actor.

In other words, once a movie is selected within the Fandango™ app, theassistant interface 110 can convert this movie information into thesemantic information interchange format language, where this translatedmovie information can then be shared with other apps such as IMDB™through the IMDB™ plugin in order to obtain additional information aboutthe movie. Such additional information to be obtained can include, forexample, which actors are playing in the movie. IMDB™ can receive andunderstand the movie selected within the Fandango™ app because thisinformation is converted into the semantic information interchangeformat language from the Fandango™ proprietary language, and then can beconverted into the proprietary language of IMDB™ by its correspondingplugin 220.

Furthermore, prior to purchasing the tickets, the user at the mobiledevice 100 may share this obtained movie information with other users atother mobile devices 100 a-n which are also connected with the assistantserver 200 (to be described in more detail with respect to FIG. 1A).Then the other users may view the obtained movie information and agreeto a certain number of tickets to be purchased, and also may ultimatelychoose another movie via the same process, at which point all users canagree to purchase tickets for a movie which all users have agreed uponafter sharing all of this information gathered between Fandango™ andIMDB™, etc., and between the users of the connected mobile devices 100a-n themselves. The above example is only an example of sharedcommunication between users and services 300 b, and is not limitedthereto. Any number of communications and any format of communicationsbetween services 300 b and users (via mobile devices 100 a-n) can beperformed through the semantic information interchange format languagecreated within the assistant server 200.

Another example of gathering information between services 300 b can berelated to music. For example, the mobile device 100 can use apps suchas Gracenote™, Shazam™, or any other type of music recognition softwareto recognize music that is playing somewhere (such as a restaurant,shopping mall, music store, radio, television, etc.). The musicrecognition can be performed by a microphone type sensor within themobile device 100 or connected to the mobile device 100. The songrecognized by the assistant interface 110 can then be converted into thesemantic information interchange format language and sent to theassistant server 200 to present this information at the Gracenote™ appthrough a Gracenote-created plugin 220. Gracenote™ can then provide thesong information to the Gracenote™ plugin 220, which in turn willprovide the song information in the semantic information interchangelanguage format to be used for any additional purpose that may berequested by the user at the mobile device 100.

In other words, at this point any additional service 300 b that knowsand/or understands how to use a song and is connected to the assistantserver 200 can use this music information (in the semantic languageformat) to provide any desired result, as may be requested by the userat the mobile device 100. For example, this information can be sent, viathe same process, to the app MusixMatch™, which presents the lyrics forthe song. It can also be sent to Rdio™, which adds the song into theRdio™ queue for the user. It can also be sent to Sonos™ speakers, whichplay the song. It can be also sent to YouTube™, which can find a videoof the song and play this video or add the video to the user's playliston YouTube™. The number of apps that this obtained and expandedinformation can be sent to in order to perform a command or gather evenmore information is only limited to the number of apps or other servicesthat can work with perform some type of function using this obtainedmusic information.

In each case, the plugin 220 for each service 300 b desired to be usedwith this obtained music information can translate to-and-from thesemantic information interchange language format and the proprietarylanguage of the service 300 b, thus providing for the free flow ofcommunication regarding the music information between all devices 300 aand/or services 300 b, as well as between multiple users. Furthermore,the music information provided in the semantic information interchangelanguage format can be sent from one user at a mobile device 100 toother users of other mobile devices 100 a-n to allow multiple users toapply this music information to various services 300 b and devices 300 aof their choosing.

In addition to the above described capabilities, if a song is playing ona television, for example, and a user of a mobile device 100 desires tocapture, save and play the song on an external portable MP3 player orother type of music playing device at a later time, the user can firstuse Gracenote™ to recognize the song playing on the television.Subsequently, this song can be translated to the semantic informationinterchange language format by the assistant interface 110 and providedto the assistant server 200, where the translated song can then be sentto a plugin 220 created for the external portable MP3 player, where thetranslated song information can be translated to the proprietarylanguage of the external portable MP3 player, where the externalportable MP3 player can then be instructed by either the plugin 220itself or the mobile device 100 to play the song at the user'sconvenience. Whether the plugin 220 is instructed to directly play thesong on the external portable MP3 player or whether the plugin 220provides the proprietary language and other necessary information backto the assistant interface 110 so that the mobile device 100 can playthe song on the external portable MP3 player can depend on either thenetwork in which the MP3 player is connected, the location of the MP3player with respect to the mobile device 100, or in accordance withother conditions as described previously herein.

In addition to providing a common set of commands, exemplary embodimentsof the present general inventive concept also allow new commands to bedeveloped, via an adaptive control system. For example, a commandreceived at the mobile device 100 which is not recognized may berecorded as a new command once defined by the user. As illustrated, forexample, in FIG. 7, a user may make a gesture intended to give a command(“Stop”, as illustrated in FIG. 7). If this input is not among theinputs recorded in the assistant server 200, the user may then input thecommand a different way, for example by touching an icon denoting“Stop.” The assistant server 200 may then record the first input (thegesture in this example) into the storage 250 as denoting the associatedcommand.

According to an exemplary embodiment of the present general inventiveconcept as illustrated in FIG. 8, natural and involuntary inputs canalso be interpreted as commands by the virtual assistant system 10. Forexample, if a user of the virtual assistant system 10 plays a video andthe audio is too loud, the user may involuntarily flinch and jerkhis/her head back. This involuntary motion will be followed by the useof icons or gestures or voice commands to lower the volume. The virtualassistant system 10 may thereby learn that the flinch and head-jerk-backgesture should be associated with lowering the volume by a large amountor muting it altogether. This new command (flinch and head-jerk-back)may be accordingly saved in the storage 250 of the assistant server 200.Similarly a grimace or other facial expression in certain contexts couldbe associated with ‘undo or reverse the last command’. Accordingly, thevirtual assistant system 10 can adapt to and store unconscious orinvoluntary input commands in addition to conscious input commands.

As an alternative, or in addition to the above exemplary embodiment ofthe present general inventive concept, a user or a developer couldspecify specific gestures which represent specific people. As anespecially contrived example, snapping one's fingers, tapping onesforehead and then gesturing to indicate a clockwise movement followed bytwo palms open to indicate ‘10’ might mean ‘send my mother a messagethat I will be there in 10 minutes.” Hence, some inputs may have a rangeof variance that represent specific things like numbers or individuals.

Some variance may also be provided in the inputs, such as the icons andgestures, themselves. Therefore, several icons may be specified in thelibrary which are mutually interchangeable and equivalent. Similarly,several gestures could correspond to the same underlying meaning and beinterchangeably used. This is important because different cultural zonesmay have varying meanings associated with specific things and havepreferences which are varying. Note that the color and size of iconscould also have meaning associated with them. Similarly, icons andgestures could take on additional meaning when used in conjunction withone another, such as denoting which device 300 a or service 300 b isbeing controlled. For example, an icon representing ‘close’ could beused to manipulate a door closed when the right palm is held flat outbut be used to close a window screen when a U shape is made with righthand instead. Such rules would also constitute a part of how the libraryis predefined and then used by plugin developers to provide users withcontrols. Note that gesture repetition may also carry meaning. Makingone knocking action with a finger versus two knocks in rapid successioncould be entirely different gestures in the gesture library. As anotherexample, for a plugin developer who is connecting the assistant to adocument management system, pinching fingers (as though holding animaginary key) and turning once might mean make a document ‘read only’whereas turning it twice could mean ‘remove both read and write access.’

Another example of such adaptive use according to an exemplaryembodiment of the present general inventive concept is illustrated inFIG. 9. As described above with respect to FIG. 2A, the mobile device100 may include a Central Processing Unit (CPU) 130. This CPU 130 may beconfigured to run a learning algorithm which determines when a series ofcommands are customarily given together. For example, a user may make afirst command, illustrated as a gesture, to raise a thermostattemperature to 70° F. The user may then make another gesture to turn onliving room lights to maximum brightness, and a third gesture to turn onthe living room television. If the CPU 130, running the learningalgorithm, determines that these gestures are customarily performedtogether, it may communicate this determination to the assistant server200, which in turn may generate a new command which executes all threeactions simultaneously. FIG. 9 illustrates is the group of commandscustomarily presented together as a separate new gesture. In otherexemplary embodiments of the present general inventive concept, multipleequivalent gestures may be proposed, any one of which will execute allthe associated actions. When one of these gestures is detected at themobile device 100, it is translated into semantic informationinterchange format language by the assistant interface 110 andtransmitted to the assistant server 200, where it is then translatedthrough the plugins 220 to give the appropriate command to thecorresponding devices 300 a 1-n and/or services 300 b 1-n.

In other exemplary embodiments of the present general inventive concept,the virtual assistant system 10, specifically the mobile device 100, maydetermine that specific icons customarily receive touch inputs at aspecific time of day, and then bring these specific icons to moreprominent positions on the user interface at that time of day. Thevirtual assistant system 10 may also present these icons (for example,as a visual depiction) on the user interface 150 of the mobile device100 in more prominent positions compared to other icons at that time ofday. The virtual assistant system 10 may alternatively generate a newicon at the assistant server 200 and displayed on the mobile device 100which executes the actions of several icons customarily receiving touchinputs together. These correlations are saved—in human readable ornon-readable format—in the storage 250 attached to the assistant server200. The virtual assistant system 10 will generate a name for this iconbased on the context. For example, an icon could be named ‘arrivedhome’.

The virtual assistant system 10 may be adaptive in another respect. If,via the CPU 130 running the learning algorithm, the mobile device 100notices that the user incorrectly but repeatedly uses an icon orsequence of icons with the intention of executing an action which failsto be executed (because the icons were incorrect) then the assistantserver 200 may actually associate that icon or icon-sequence with thataction (for that user, for a subset of users, or for all users). Forexample, if a user repeatedly tends to turn up the volume and thenimmediately turns it back down and then (using a different icon) movesup a channel of a television, this indicates that the user is confusingthe icon for turning up the volume with channel-up. The CPU 130 in themobile device 100 can detect this behavior and then propose that the“volume-up” icon be used for “channel-up” instead and in turn bereplaced with a different and less-confusing icon. The virtual assistantsystem 10 can also automatically make the change if it deems itappropriate based on a decision-making algorithm that is running in theCPU 130, the assistant server platform 210, or on a separate serverwhose results are saved on the storage unit 140 or storage 250.

These usage patterns and the accompanying changes may be collected inthe storage unit 140 of the mobile device 100 or in the storage 250 ofthe assistant server 200 to be used as updated commands. As a result,the library of commands 230 stored on the storage 250 may expand overtime with new, intuitive and adaptive commands.

Alternatively, or in addition to the above adaptive control in which theassistant server 200 automatically updates the commands, data on theuser's actions and inputs may be collected and made available to theplugin developer, for example, by e-mail or an online administrativeportal through which the developer can view information related to theusage of the plugin 220. This allows the developer to update the plugin220 accordingly. This is valuable because the repeated use of theinputs, for example icons, indicates a persistent intuitive mapping ofthem and the intended action in the mind of that user or in an aggregategroup of users. The developer can discover that such a psychologicalinput-to-action mapping exists based on user behavior. Separatelytransmitting the data to the developer instead of implementing alearning algorithm in a CPU can be advantageous when processor capacityor the effectiveness of a particular learning algorithm is a concern.

These adaptive techniques may be employed to address semantic confusionbetween plugins 220 created by different plugin developers. For example,the developer of the plugin 220 for one thermostat may specify onegesture to raise temperature whereas the developer of the plugin for adifferent thermostat may specify a different one. Based on how users areemploying gestures, the virtual assistant system 10 may providesuggestions to the developer on how gestures (or icons or voicecommands) can be modified to allow users to accomplish what they wantwith a more consistent set of gestures. By classifying services intosimilar groups, developers will also be helped to select the mostintuitive icons, gestures and voice commands for their new plugins 220.For example, a developer creating the plugin 220 for the tenththermostat to interface with the virtual assistant system 10 may be ableto be better informed by how users actually prefer to invoke variousthermostat functions by means of icons, gestures and voice with theprevious nine thermostats. This could be done prescriptively by stronglysuggesting the developer use specific icons for specific functions ormore indirectly by discouraging mappings which have been shown to beconfusing for users in the past.

Furthermore, these adaptive techniques can be employed using variousartificial intelligence techniques, which are to be utilized by and/ordeveloped on various types of hardware, including robotics algorithms,learning algorithms, parallel programming, logical searching,optimization, heuristics, evolutionary computation, etc., but are notlimited thereto. Accordingly, the virtual assistant system 10 can“learn” a user's preferences based on the user's movements, habits, andtendencies.

According to another exemplary embodiment of the present generalinventive concept, different sets of gestures, icons and voiced commandscan become established for different groups of users (e.g. in differentcountries). The virtual assistant system 10 (either the CPU 130 of themobile device 100 or the assistant server platform 210 of the assistantserver 200) can infer from the use of specific inputs (gestures, voicecommands, icons, etc.) which of these groups the user belongs to andthen provide support to them in the “dialects” that user is mostfamiliar with. For example, it is possible that people coming frommilitary contexts get used to a different set of icons or gestures tomean “turn left” than people coming from civilian contexts. If thevirtual assistant system 10 is able to infer that someone ismilitary-affiliated based on a gesture the person makes to indicate“turn left,” other gestures and icons may also change meaning to providethis person with a more familiar context. Such inferences can then besaved in the virtual assistant system 10 as a context of this person(specifically, in the storage unit 140 of the mobile device 100 and/orthe storage 250 of the assistant server 200).

According to another exemplary embodiment of the present generalinventive concept, there may be a three tier system of libraries. Thefirst set of (gesture, voice, icon) libraries are universal and apply toall users. The second set of (gesture, voice, icon) libraries apply tospecific cohorts of users (e.g. those in a specific trade, those of acertain age or social grouping, those who speak a given language, thosewho are members of a specific set of institutions, those who reside inspecific regions or countries, etc.). The third set of (gesture, voice,icon) libraries apply to specific users. If there is conflict betweenthese because the same words, gestures or icons mean different things inthese three different tiers of libraries, a precedence rule can beutilized. Normally, user-specific interpretations trump user-cohortinterpretations which in turn trump universal interpretations. However,this precedence order can be different in some situations.

Controls may also be automated or vary depending on context. Forexample, if the user always turns on the DVR after turning on the TV,the virtual assistant system 10 can automatically choose to turn on theTV when receiving a command to turn on the DVR. Alternatively, thispredictive behavior may be programmed by the plugin developers accordingto use or context data gathered by the virtual assistant system 10. Asanother example, access to a certain document management system canautomatically be enabled if a user is present in a particular geo-fencedarea, and such access may be cut off when the user exits that area.Control can be made even more specific, e.g., as illustrated in FIG. 10,a user that enters a room through one corridor they might have access tothe document management system, whereas if the same user enters from adifferent corridor that user may not have access. Other factors, forexample, time or the sequence of user usages of the virtual assistantsystem 10 can also be an element of context, thereby varying thecontrols of the virtual assistant system 10.

In order to enable these scenarios where the virtual assistant system 10takes predictive action, according to an exemplary embodiment of thepresent general inventive concept of the virtual assistant system 10,the mobile device 100 or the assistant server 200 can constantly measurewhen certain actions are invoked and collect these measurements in itsstorage (storage unit 140 of the mobile device 100 and/or the storage250 of the assistant server 200). The measurements could includelocations, times of day, co-presence of specific individuals, actionsequences that have been taken and other factors. If the virtualassistant system 10 finds a reliable way to predict when certain actionsare taken based on statistical, probabilistic or machine-learningalgorithms that are running on the CPU 130, the assistant serverplatform 210, or on servers or other learning type devices connected toor in communication with the virtual assistant system 10, then thevirtual assistant system 10 can do so.

The virtual assistant system 10 can also proactively present itsintention to predictively take such actions to the end user by means ofdisplaying a message on the mobile device 100. This message can betransmitted from the assistant server 200 to the mobile device 100 to bedisplayed on the UI 150, or the message can be generated at the CPU 130and displayed at the UI 150. The message may be visual, audible, ortactile, and allow the user to approve or disapprove such actions. Themessage can take the form of presenting the predictive action in advanceas a new “standing rule” or proposing that the virtual assistant system10 take the action momentarily unless the user overrides the proposaland instructs the virtual assistant system 10 not to.

As will now be described in more detail while referring to FIG. 1A andFIG. 11, a user of the mobile device 100 may share access to theassistant server 200 (hence the virtual assistant system 10) to allowother users to control devices 300 a and services 300 b within thevirtual assistant system 10 and therefore already being controlled bythe user sharing the access. Thus, a user (at mobile device 100) canshare the semantic information interchange format language with otherusers through the other users' respective mobile devices 100 a-n.Accordingly, any desired control of a device 300 a or service 300 b bythe user of the mobile device 100 can be shared with another userthrough the other user's mobile device 100 a (see dotted lines “sharingaccess”) by “friending” the other user's mobile device 100 a, asillustrated in FIG. 1A. This sharing (or “friending”) of access tocontrol devices 300 a and services 300 b can be provided once the user(mobile device 100) has a way of contacting the other user, for example,via email, LinkedIn™, Facebook™, Twitter™, etc. In other words, once auser of the mobile device 100 within the virtual assistant system 10 isable to communicate with another user via another user's mobile device100 a-n through any form of communication, the user (mobile device 100)can send an “offer” to share access to the assistant server 200 with theother users' mobile devices 100 a-n via this form of communication.

Once the other user(s) accepts the offer of access to the user'sassistant server 200, the other user(s) may be provided with the samecontrol over all external devices 300 a and services 300 b as the userof the mobile device 100 possesses through the assistant server 200.Moreover, as described in more detail below, the shared access withother users may be limited in many aspects, such as, for example, withrespect to the level of control, the location in which the other user(s)is situated, a time frame in which the user allows other user(s) tomaintain the shared access, etc. There are a vast number of ways inwhich access sharing to other users may be controlled/limited based onthe desired setup of the virtual assistant system 10 by either thedeveloper of the system 10 or the user's instructions at a later timeafter the original setup of the system 10.

The shared access allows the other users to control external devices 300a and services 300 b through the original user's assistant server 200and to the level at which the original user allows the shared (or“friended”) user to have access.

Furthermore, as illustrated in FIG. 11, at #1, although the originaluser A (the user making the offer) must initially provide ausername/security ID and password (or “access credentials”) in order toaccess an external device 300 a or service 300 b, the user(s) B whobecomes “friended” (receive shared access) does not need to enter thissecurity ID or password. Therefore the original user A is never requiredto share these access credentials with the other user(s) B being“friended.” Accordingly, when user A instructs the assistant server 200to allow access to a device 300 a or service 300 b (see #2 of FIG. 11),the user B receiving the shared access can then control the device 300 awithout knowing the access credentials input by user A, as pointed outat #3 of FIG. 11. Thus, no access credentials need to be exposed orshared with any other users in order to “friend” another user(s) andallow the other user(s) to have a full or limited access to any device300 a or service 300 b, or even the assistant server 200. Furthermore,the original user A can instruct the assistant server 200 to revoke userB's access to the device 300 a or service 300 b at any time, oralternatively can declare specific conditions under which revocation ofaccess to other users, i.e., user B, can occur.

In addition to keeping the ID and password (or other credentials)confidential from other users, since the ID and password remain only atthe assistant server 200, and therefore never need to be communicatedwirelessly, the user's IDs and passwords (or other credentials) cansafely remain confidential to the original user A (at mobile device100). Accordingly, illegal access, impermissible access and misuse ofsuch credentials via a wireless transmission interception or other typeof compromised security can be entirely prevented.

Thus, the assistant server 200 maintains all credentials (ID, passwords,etc.) that give authority to access and use the underlying services 300b or devices 300 a. These credentials are maintained by the assistantserver 200 at the storage 250 that is connected either directly orindirectly to the assistant server 200. When a plugin 220 is created atthe assistant server 200, and a user begins to access this plugin 220for the first time in order to have the assistant server 200 carry outvarious actions with the corresponding service 300 b, the user firstneeds to provide the required credentials for the service 300 b to theassistant server 200 in some form. These credentials can be collected bythe assistant server 200 in one of many means: through a screen,keyboard, scanner, mouse, camera, microphone or hand-held or wearabledevice (such as armband or facewear or fingerwear, etc.) or anotherperipheral hardware system or systems connected to or in communicationwith the assistant server 200.

As an example, the user A (at mobile device 100) may provide a usernameand password for an email service to the assistant server 200. Now theassistant server 200 becomes capable of accessing and using the servicevia the Internet, Bluetooth, Optical or visual signaling, audiosignaling (through an audio speaker or other mechanism), tactile orother mechanical signaling using a device such as a vibration unit,WiFi, USB, FireWire, HDMI or through other means on the user's behalf.The user can now instruct the assistant server 200 to share access tothe service 300 b with another user(s) B as well. As described above,this instruction by user A enables another user(s) B to access theservice 300 b via the assistant server 200, and only through theassistant server 200. As far as the underlying service 300 b (i.e.,email) is concerned, it is being accessed by the original user (user A)either through the assistant server 200 or through the mobile device 100if the assistant server 200 and the mobile device 100 are providedtogether in one hardware, and only the assistant server 200 is awarethat the service 300 b (or device 300 a) is, in fact, being accessed byanother user (user B).

Another example of this secured form of sharing is described as follows,while still referring to FIG. 11: the original user A establishes adevice 300 a in the assistant server 200 using the assistant interface110, e.g., the original user A may establish a certain set of lights asan external device 300 a to be accessed and controlled wirelessly. Theuser may then provide the assistant server 200 necessary informationrequired to access and operate the lights. This information couldinclude, for example, a password, username, and an IP address of thelights. The plugin engineer initially specifies which credentials areneeded for a user to start using that type of external device 300 a withthe assistant server 200. Now the lights exist as the external device300 a in which the assistant server 200 can access. This access controlis owned only by that user A. This user A can now instruct the assistantserver 200 to “friend” those lights (device 300 a) to any other user Bas desired. This sharing of access can be accomplished in variousdifferent ways as described herein. For example, user A may instruct theassistant server 200 the share access by contacting user B and offeringuser B to join the virtual assistant system 10, or can make a gesturetoward the user B if user B is within an area in which a gesture cansignal user B, or user A can make a gesture that signifies allowingaccess, at which point a camera detects a user B in the vicinity, etc.The other user(s) B can now see the external device 300 a via theassistant server 200 and use the external device 300 a. The other user Bcan, for example, turn these lights on and off, and change the color ofthese lights. However, as described above, this other user B does notpossess the username, password or IP address (or other credentials asoriginally created by the engineer) of the lights and has no means ofoperating the external device 300 a except through the assistant server200.

At any time the owner user A can instruct the assistant server 200 to“unfriend” the friended user B (revoke access) from the external device300 a. If the owner user A revokes access to the other user(s) B, theother user(s) B ceases to be able to operate the lights or, in somecases, user B may even be blocked from being able to see these lights inany fashion at all.

A user may also set criteria by which “unfriending” occursautomatically. Such criteria can be a time limit of access shared withuser B, a location condition placed on user B, and so on. For example,services may be shared with the other user B for a preset amount oftime, or until the other user leaves a particular area.

There are an extensive number of other applications that can beimplemented by the virtual assistant system 10. Controlling a set ofhome devices 300 a are one obvious application. If someone visits auser's home, the host can instruct the virtual assistant system 10 togive them courtesy access to all or a large subset of the connecteddevices 300 a (and/or services 300 b) in the user's home (lights,thermostat, window blinds, alarm system, phone system, WiFi hub,entertainment systems, etc.) as long as the users being “friended” arewithin the physical confines of the home. This could also be done basedon calendar invites. In other words, along with inviting others to auser's home, the user A owning the home could also create a formalinvite to the users to be “friended” which specifies access to homefacilities during the time of the visit.

This limited access sharing can also be implemented in a businesscontext. When someone joins a team, the virtual assistant system 10 canextend access to various tools and services to this person via theassistant server 200. When this person ceases being part of the team,this access can be revoked. This revocation can be performed by simplyrequesting a list of those members provided with access, and thenselecting among certain members in which the user A requests access tobe revoked.

The assistant server 200 can also provide this access based on paymentsbeing made. For example, a visitor to a movie theater can have access toa soda fountain via the theater's assistant server 200 for as long as amovie is playing. This time based access sharing can be automaticallyrevoked at any designated time, i.e., at completion of the movie.

As another example, a user A (of mobile device 100) can shareinformation from all his/her health devices with his/her doctor. Thedoctor, being user B, can be provided access to all health informationstored in the storage 250 of the assistant server 200.

As described above, the shared access offered by the initiating user Aof the mobile device 100 to other users B can be completely controlledby the user A of the initiating mobile device 100 such that access byany of the shared access users B can be terminated at any time by theowner user A, as well as being limited to a certain level of control, acertain time period of control, control limited to within a certainarea/location, etc.

Sharing can also be predictive in nature, where the predictive rules andalgorithms may or may not be humanly readable, and are saved in astorage 250 connected to or in communication with the assistant server200, where they are running. Alternatively, as pointed out above, undercertain situations the predictive rules and algorithms related to theaccess sharing can be running on the mobile device 100, and thereforestored in the storage unit 140. For example, if a certain set ofcontexts usually result in a user A sharing access to some services withanother user B, the assistant server 200 may recognize this contextualpattern and in the future perform this sharing automatically. Theassistant server 200 may also present the owner user A with the choiceas to whether the contextual pattern should be determined and appliedautomatically, or whether the user A should make this decision. Forexample, the assistant server 200 can alert the user A that theassistant server 200 intends to share access to the lights or otherdevices 300 a (or services 300 b) based on a recognized contextualhistoric pattern, such as sharing a thermostat control, a television,etc., with a guest user B unless the user A explicitly wants to refusesuch access. At this point the user A can select an option provided bythe assistant server 200 by touching an icon on the display 150, makinga gesture to a sensor 160 within the mobile device 100, and provide aninput command via another form of input as described herein.

As pointed out above, sharing is also possible to perform at varyinglevels. For example, an online commerce service like Amazon™ hasmultiple things one can do with it. Ratings and pricing of various itemscan be reviewed in a personalized way (“this is the rating for otherreaders like you” or “as a prime member, you don't pay for shipping forthis item”), items can be added to a wish list, items can be added tothe Amazon™ shopping cart and purchases can be made. It is possible tocreate subset services where items can be reviewed and placed in theAmazon™ shopping cart but purchases cannot be completed. It is possibleto see how a parent (user A) can provide their children (user B) withsharing an Amazon™ account to the extent of putting things in theAmazon™ shopping cart but not actually making the purchase. This way theparent (user A) can review all items the children (user B) want beforethe purchase is made. In other words, access sharing can apply to evensubsets of a plugin's functionality rather than all of itsfunctionality. Similarly, other rules can also be crafted. A parent canlimit his/her children to change TV channels within certain values,while the parent (User A) can go to any channel desired via theassistant server 200.

This limited and context-based sharing has other applications also. Forexample, a city can allow an average citizen to operate doors or othersystems in emergency situations where it is normally prohibited to them,and can only be operated by a specialist or normally-authorizedpersonnel. Access can also be extended to specific groups of people.Based on these people being added to a specific user group, pregnantwomen (in a business, hospital, shopping mall or community) may be givenaccess to certain services 300 b which are inaccessible to others forthe duration of their pregnancy. Users whose electric cars break down ina specific way may have access to loaner cars in unmanned car parkswhere any normal user does not. Access to the user's broken-down car maypredictively be granted to a repairman who heads out to the car. Here,the users A, via the mobile device 100, will extend specific permissionsto the repairman's mobile device 100 a.

Social, family and community connections will be inferable from sharingpatterns as well. This too can be used to make predictive sharinghappen. If all home devices are being shared by one user A with anotheruser(s) B, it is likely that the next home device to be registered atuser A's home will also be shared with the same other user(s) B. Suchpatterns will be saved in the storage 250 and will be inferred usinganalytic algorithms created within the assistant server platform 210 orobtained as patterns that can be imported into the assistant serverplatform 210 from other software programs running on other servers.

Note that users can quickly forget their usernames and passwords tovarious services where access via the assistant server 200 suffices. Anowning user A of mobile device 100 can also designate another user B ashaving the power to share access with yet other users. This can beimplemented, for example, among members of a single household. One usercan purchase a TV and interface it with the assistant server 200, andthen provide access rights to other family members (users B), who canthen each extend access to this TV (device 300 a) as-needed to theirrespective guests.

The assistant server 200 can also transfer ownership of services fromone user A to another. This allows, for example, user A to sell theirhouse to another person and transfer all devices 300 a within the houseto that person. Similarly, a person (user A) changing offices cantransfer ownership of all devices 300 a and services 300 b fromthemselves to another user B, where user A can then take ownership ofother devices 300 a and services 300 b associated with a new office inwhich user A moves into. As pointed out above, services can be grantedbased on certain conditions also. For example, if a person unfortunatelydeceases, then their devices 300 a and services 300 b can automaticallybe turned over to their relevant family members or other desired people.

As pointed out above, gestures are a form of commands used with thevirtual assistant system 10. Gestures are an easy way to operate theauthority sharing features described above. The use ofgestures—including eye movement or non-movement, eye brow movement, agaze and a facial expression and all other visually-detected bodymovements (or non-movement) to extend or revoke authority of usage tousers or groups of users—can be quite powerful, and can be performedusing the virtual assistant system 10 as described herein.

Referring to FIG. 12, sharing access to control of a set of lights(external device 300 a) to a user B could be as simple as the owner userA pointing to the lights and then making a gesture representing sharing(see Time #1) this access with the other user B. As a result, theassistant server 200 can grant access to user B. More specifically, insuch an environment where a user A regularly works or lives, forexample, a set of sensors, such as a camera or other type ofphoto-detecting device that are capable of capturing information about auser's movements, can be set up within the controlled environment. Thesesensors can be configured to have their own form of an assistantinterface 110 a-n such that when these sensors (see FIG. 12) detect agesture by user A to share access to control devices 300 a such as lightbulbs, etc., within the controlled environment, this gesture command canbe translated at the relevant sensor into the semantic informationinterchange format language and provided directly to the assistantserver 200. As a result, the assistant server 200 can share access witha user B as instructed by the user A via a gesture command inputrepresenting sharing access.

For higher security situations, a camera or other type image sensor orother type sensor may be able to detect the actual hand or fingerprintsof the user A in order to determine that it is in fact user A that isinstructing the assistant server 200 to share access to another user B.Alternatively, the user A may have a wearable type device oraccelerometer that detects a gesture made by user A. Moreover, when thesensor detects user B via the camera or other type recognition device,the assistant server 200 can store any predetermined detected charactertrait of user B and store this information within the storage 250 sothat shared access may be provided to user B during later visits to suchan environment.

Still referring to FIG. 12, revoking control to a user B can be assimple as making a gesture for “unshare” towards user B (see Time #2),which leads the assistant server 200 to revoke access to the device touser B. Alternatively, revoking can be performed by other gestures, suchas, for example, making the same set of gestures as those of sharingaccess to user B, and then making a chopping gesture. This is merely anexample to illustrate the various types of gestures that can beprogrammed into or learned by the assistant server 200. These gesturesby user A can also be captured by a camera or other photo-detecting orsensing device that is capable of capturing information about movementsof user A, which is in turn connected to the user's mobile device 100.Such gestures may also be captured by wearable, held or other devicesthat detect movement by using accelerometer hardware or othermovement-tracking hardware, or even eye-tracking, lip-tracking orfacial-tracking systems.

Alternatively, as described previously, user A may request a recent listof users B in which access has been shared. Once this list is providedto user A at the user interface 150 of mobile device 100 by theassistant server 200, user A may choose any user(s) B on the listprovided at the user interface 150 and instruct the assistant server 200through the user interface 150 to revoke future access.

It is also possible to set-up rules where access is extended temporarilyto another user B and then must be extended periodically (or upon theoccurrence of an event, such as someone departing from a geo-fencedarea) with the use of a gesture. In this type of access sharing, if theowning user A makes such a gesture then access can be continued. If theowning user A fails to make the gesture, then access lapses.

Gestures may also be involuntary or be made in a way of which a user Bis unaware. For example, a commercial establishment where products aresold may make door locks to certain restrooms operate for customers orprospective customers (i.e., user(s) B) who spend specific amounts oftime in certain areas of the establishment, but not to others. Such acommercial establishment may even allow goods contained in securedclosets or display cases to become accessible to certain customers basedon gestures and movement information, which are not available to others,by granting them access to open the securing enclosures where others arenot granted such access.

In geo-fencing and other such applications, locations can be determinedby means of a GPS unit, WiFi unit, Bluetooth facility,location-broadcasting buoys, IP addresses, NFC, etc., which areconnected to or in communication with the assistant server 200.Proximity-detection to location measuring sensing devices which may beembedded in automobiles, household objects, office objects and otherobjects can also measure location. Time can also be an element of theaccess determining means which can be setup and measured by a similarsuch computing device. The determining means of access can also includethe sequence of user usages of the assistant server 200.

The virtual assistant system 10 described herein also provides theability for users to have secret gestures for revoking and grantingaccess to control external devices 300 a and services 300 b. Forexample, in a home break-in situation, a parent User A may grant theright to a child to summon the police using the home alarm via a secretgesture (where the child may not normally have this right).

These gestures may also operate in conjunction with icon-touches andvoiced commands. The user A, for instance, might make a gesture thatmeans ‘share’, then user A may recite the names of all the serviceshe/she intends to share. A voice sensing device can detect the namedservices and accordingly provide access for these services. Then user Acan point at the receiving user B as described above.

These gestures may also be used in situations where a subset of usersfrom a large subset of users have to be granted or ungranted access to aservice (or set of services). An example of such a situation is where adecision needs to be made as to who has access to a given cash registerin a store from among a hundred sales associates. In such a situation,simply laying out a list of names or photos of the associates andpointing at the particular ones to whom access should be granted can beone mechanism for granting a subset of users particular access. Anothermechanism can be through gaze-tracking where the user A looks at theparticular ones who should be granted provisional access longer than theamount of time user A looks at the ones who should not be grantedaccess. Such gestures can be conscious or unconscious on the part ofuser A.

Similarly, a situation can occur in the reverse where there are a largeset of services and a decision needs to be rapidly made as to whichservices access should be provided (or access revoked from) for a useror a set of users. Gestures by the user A—conscious or unconscious—canbe used in this situation as well.

Gestures may also be used to seek access to services 300 b or devices300 a by users seeking shared access (i.e., a user B seeking access inwhich the user A with access rights). This enables a “silent”conversation where one user seeks access to a given service 300 b ordevice 300 a with one set of gestures and is granted such access—eitherby an owner user A with another set of gestures or through some decisionmade programmatically (algorithmically) by the assistant server 200 andsensor device combination which is observing the gestured request. Thesealgorithms that determine such shared access can be software programs orparts of software programs running as part of the assistant server 200together with other programs running within the assistant serverplatform 210.

Referring to FIG. 13, a user of the mobile device 100 can add, includinga mass-addition, services to an assistant server 200 based on aquestionnaire, survey or form-filling of credentials and otherservice-related data, as well as based on discovery.

More specifically, it is likely that when the assistant server 200becomes established other users will want to provide all theirindividual account credentials to the assistant server 200 so the otherusers can rapidly establish connections to those services 300 b. Forexample, a user can give his/her usernames and passwords to LinkedIn™,Yahoo™, Google™, Amazon™, Facebook™, etc., to the assistant server 200at one sitting, which enables the assistant server 200 to systematicallyconnect to these services 300 b all quickly. This can be done by fillingin a form that the assistant server 200 provides to the users. This formcan be a spreadsheet, email form, fillable PDF, web-based form, avoice-based question and response session, or any other data collectingmechanism through which this type of information can be collected. Theform can be delivered to the user via the Internet or via some otherconnection mechanism such as Bluetooth or NFC, or even physical mail orcourier service.

The assistant server 200 can also take the user through a process wherethe user is asked for credentials to most common services 300 b that theuser is likely to be using—which is similar to filling out a survey. Theassistant server 200 can do something similar for devices 300 a. Forexample, the assistant server 200 can detect devices 300 a on the user'shome network and then connect to them automatically by requesting thecorresponding credentials when necessary. Information relating to thesecredentials and devices 300 a can be saved on a storage 250 that isconnected to the assistant server 200.

Similarly, the assistant server 200 can actually take the user's emailsand then probe most common services 300 b 1-n with those emails to seeif they already have associated accounts. In this case, the user canprovide various email addresses to the assistant server 200, which cansave these email addresses and then use them to attempt to log intovarious web services and other services. If these users have forgottentheir passwords, the assistant server 200 can initiate a password resetprocess (i.e., either by conducting the password reset processautomatically or with the user's help if services like Recaptcha™ areinvolved, which require humans to recognize obfuscated letters andnumbers) and revive those accounts and link them to the user's accounts.This also means that users may no longer have to come up with passwords.The assistant server 200 can come up with highly complex passwords whichthe user does not need to remember since the users are almost alwaysgoing to access the underlying services through the assistant server200. These highly complex passwords are then saved on the storage 250.

The assistant server 200 can also take a user through a process ofestablishing brand new accounts with various services 300 b 1-n that arerelevant to people similar to the user. This can have many applications.For example, the assistant server 200 can be used to establish a votingaccount in an election system for a citizen when they achieve a certainage. It can be used to enable the citizen to establish an account withbmw.com when they buy a BMW. The rules determining such new servicesetups can either be saved on the storage 250 or be communicated to theassistant server 200 by other services 300 b 1-n that are able tocommunicate with the assistant server 200 over the Internet, WiFi,Bluetooth, NFC, USB or other mechanisms.

The present general inventive concept can also be embodied ascomputer-readable codes on a computer-readable medium. Thecomputer-readable medium can include a computer-readable recordingmedium and a computer-readable transmission medium. Thecomputer-readable recording medium is any data storage device that canstore data as a program which can be thereafter read by a computersystem. Examples of the computer-readable recording medium include asemiconductor memory, a read-only memory (ROM), a random-access memory(RAM), a USB memory, a memory card, a blue-ray disc, CD-ROMs, magnetictapes, floppy disks, and optical data storage devices. Thecomputer-readable recording medium can also be distributed over networkcoupled computer systems so that the computer-readable code is storedand executed in a distributed fashion. The computer-readabletransmission medium can transmit carrier waves or signals (e.g., wiredor wireless data transmission through the Internet). Also, functionalprograms, codes, and code segments to accomplish the present generalinventive concept can be easily construed by programmers skilled in theart to which the present general inventive concept pertains.

Although a few embodiments of the present general inventive concept havebeen shown and described, it will be appreciated by those skilled in theart that changes may be made in these embodiments without departing fromthe principles and spirit of the general inventive concept, the scope ofwhich is defined in the appended claims and their equivalents.

What is claimed is:
 1. A method of controlling an external service amonga plurality of external services, the method comprising: receiving aninput command at a mobile device, the input command corresponding to afunction to be performed at the external service; translating the inputcommand into a semantic information interchange format language at themobile device; receiving the translated input command at a server devicewirelessly connected to the plurality of external services; obtaining aproprietary language of the external service to be controlled at theserver device, the obtained proprietary language corresponding to thetranslated input command; and determining whether to perform thefunction at the external service using the obtained proprietary languageat one of the server device or the mobile device based on predeterminedconditions.
 2. The method of claim 1, further comprising: performing thefunction at the external device by the mobile device when it isdetermined that a local network connection is established between themobile device and the external service, otherwise performing thefunction at the external device by the server device through thewireless connection.
 3. The method of claim 2, wherein if it isdetermined that a local network connection is established between themobile device and the external service, further comprising: translatingthe obtained proprietary language of the input command into the semanticinformation interchange format language at the server device;transmitting the translated proprietary language to the mobile device;and converting the transmitted semantic information interchange formatlanguage input command back to the proprietary language at the mobiledevice.
 4. The method of claim 1, further comprising: analyzing contextinformation of the input command at the server device to obtain historyof use data of the input command.
 5. The method of claim 4, wherein thecontext information includes at least one of a location of the mobiledevice when the input command is received, a time of day when the inputcommand is received, co-presence of specific individuals, involuntaryactions by a user, and action sequences taken by a user of the mobiledevice.
 6. The method of claim 4, wherein: the history of use dataidentifies at least one specific user; and the received input command isconverted according to the at least one specific identified user.
 7. Themethod of claim 4, further comprising: authorizing at least one otheruser to control the external service through input commands according tothe history of use data, wherein the history of use data comprises dataidentifying a history of providing authorization to other users tocontrol the external service through input commands according to storedcredentials.
 8. The method of claim 1, wherein if it is determined thatthere are a plurality of possible intended functions to be performed bythe input command at the server device, further comprising: displayingthe possible intended functions to be performed at the mobile device tobe selected as a confirmation input command; and updating a storedlibrary of recognizable input commands according to the selectedconfirmation input command received.
 9. The method of claim 1, furthercomprising: recognizing a repeated cluster of input commands at theserver device; and creating a new consolidated input command within astored library of semantic information interchange format commands torepresent performing the cluster of input commands simultaneously. 10.The method of claim 9, wherein: the cluster of commands comprises afirst input command that is invalidated and a second input commandsubsequently input; and updating the library comprises associating thesecond input command with the function previously associated with thefirst input command.
 11. The method of claim 1, further comprising: ifthe input command is not recognized, requesting an alternate inputcommand at the mobile device; and updating a stored library of semanticinformation interchange format commands with the received alternateinput command.
 12. The method of claim 1, further comprising: storingcredentials at the server device, the credentials corresponding to auser and including data to authorize the user to control the externalservice through the input commands; and updating the stored credentialsat the server device according to a received input command to authorizeat least one other user to control the external service through inputcommands.
 13. The method of claim 12, further comprising: revoking theauthorization of the at least one other user to control the externalservice at the server device according to another input command.
 14. Themethod of claim 13, wherein the receiving the other input command torevoke the authorization includes: displaying a list of authorized usersat the mobile device; and selecting the at least one other user to haveaccess revoked.
 15. The method of claim 12, further comprising:connecting the server device to the plurality of external services bypresenting the stored credentials to the respective external servicesand determining whether further authentication is necessary to inputcommands to the services; displaying a request for input of additionalcredentials if it is determined that further authentication isnecessary; and updating the stored credentials with the additionalcredentials received.
 16. A non-transitory computer-readable mediumcontaining computer-readable codes to perform the method of claim
 1. 17.A method of controlling an external service among a plurality ofexternal services, the method comprising: storing credentials of a userat a server device; receiving at a mobile device an input command toperform a function at the external service to be controlled; translatingthe received input command to a semantic information interchange formatlanguage at the mobile device; determining at the server device whetherthe stored credentials authorize the user to input commands to controlthe external service; converting, at the server device, the translatedinput command to a corresponding input command understood by theexternal service when the credentials authorize the user to inputcommands; and controlling the external service with the input commandunderstood by the external service.
 18. The method of claim 17, whereinwhen it is determined that a local network connection is establishedbetween the mobile device and the external service, controlling theexternal service with the input command understood by the externalservice comprises: transmitting the input command understood by theexternal service to the mobile device; and controlling the externalservice from the mobile device by transmitting the input commandunderstood by the external service from the mobile device to theexternal service.
 19. The method of claim 17, wherein when it isdetermined that a local network connection is not established betweenthe mobile device and the external service, controlling the externalservice with the input command understood by the external servicecomprises: controlling the external service from the server device bytransmitting the input command understood by the external service fromthe server device to the external service.
 20. A non-transitorycomputer-readable medium containing computer-readable codes to performthe method of claim
 17. 21. A method of controlling an external serviceamong a plurality of external services, the method comprising: storing alibrary of commands in a semantic information interchange formatlanguage in a storage device, each command corresponding to a respectivefunction to be performed at one of the plurality of external services;receiving an input command at a mobile device; translating the inputcommand into the semantic information interchange format language at themobile device; transmitting the translated command from the mobiledevice; receiving the translated command at a server device; determiningat the server device if the translated input command is included in thelibrary of commands; and if the translated input command is included inthe library of commands, controlling the external service at one of theserver device and the mobile device according to a proprietary languageof the external service to perform the associated function.
 22. Themethod of claim 21, wherein the library of commands includes a set ofcommands associated with a specific user.
 23. A non-transitorycomputer-readable medium containing computer-readable codes to performthe method of claim
 21. 24. A method of communication between services,the method comprising: receiving at a server device an output data froma first external service expressed in a proprietary language of thefirst external service; converting the received output data to asemantic information interchange format language at the server device;converting the converted output data from the sematic informationinterchange format language to a proprietary language of a secondexternal service at the server device; and transmitting the convertedoutput data in the proprietary language of the second external device tothe second external service as an input command to perform acorresponding function at the second external device.
 25. Anon-transitory computer-readable medium containing computer-readablecodes to perform the method of claim
 24. 26. A method of controlling aplurality of external services within a virtual assistant system, themethod comprising: translating an input command at a mobile device intoa semantic information interchange language format that can becommunicated between the plurality of external services, the inputcommand representing a function to be performed at one of the externalservices; transmitting the translated input command from the mobiledevice; receiving the transmitted input command at a server device;determining, at the server device, whether the mobile device or theserver device will perform the function at the external service to becontrolled based on at least one of a location of the mobile device withrespect to the external service to be controlled and a networkcommunication established between the mobile device and the externalservice to be controlled, and attaching proprietary language of theexternal service to be controlled to the received input command toperform the associated function by either the mobile device or theserver device according to the determination result.
 27. The method ofclaim 26, wherein the proprietary language of the corresponding externalservice is attached to the received input command by embedding theproprietary language therein to enable communication with the externalservice to be controlled by the server device or to be transmitted backto the mobile device to directly control the external service.
 28. Amethod of controlling an external service from among a plurality ofexternal services connected to a virtual assistant system, the methodcomprising: receiving an input command at a mobile device, the inputcommand corresponding to a function to be performed at the externalservice; translating the received input command into a semanticinformation interchange format language at the mobile device;transmitting the translated input command from the mobile device to aserver device; determining a location of the mobile device at the serverdevice; determining the function to be performed at the externalservice, the determination being based on the determined location of themobile device; and performing the determined function at the externalservice by converting the translated input command at one of the mobiledevice and the server device into a command recognizable by the externalservice to perform the determined function.
 29. The method of claim 28,further comprising: determining at the server device whether to convertthe translated input command at the mobile device or the server devicebased on the determined location of the mobile device.
 30. The method ofclaim 29, wherein if the determined location of the mobile device iswithin a predetermined distance of the external service, determining atthe server device to convert the translated input command at the mobiledevice, otherwise determining at the server device to convert thetranslated input command at the server device.
 31. The method of claim28, wherein the converting the translated input command comprisesdetermining if a user of the mobile device is authorized to inputcommands to the external service, based on at least one of thedetermined location of the mobile device and whether there arecredentials corresponding to the user stored at the server deviceauthorizing the user to input commands to the external service.
 32. Themethod of claim 31, wherein if the determined location of the mobiledevice is within a predetermined geo-fenced area, further comprising:determining that the user of the mobile device is authorized to inputcommands to the external service.
 33. The method of claim 28, furthercomprising: receiving an input command at the mobile device to controlthe server device to determine the location of another mobile device,and to authorize the other mobile device to input commands to theexternal service while the other mobile device is within a preset area.34. A non-transitory computer-readable medium containingcomputer-readable codes to perform the method of claim 28.